The Deterrence Lockbox
QUOTE
Various Authors
calendar-icon_whitecircle
Deterrence in the Wild:
Part 2
In the words of Helen Keller, "Alone we can do so little; together we can do so much."

To help further our dialogue on deterrence and better understand how it functions in the assessment industry, we asked a few respected test security experts about their personal experiences with deterrence. How have they utilized it in their testing programs, and what value have they received in return?

Let's draw on the insight and wisdom of our colleagues as we work together in pursuit of increased test security, fairness, and exam validity.
You might also like...
More Reads
Get to know your colleagues in the test security industry a little bit better as they answer fun and intriguing questions from Marcel Proust's 18th-century parlor game.
Perhaps the most compelling way to deter potential wrongdoers is to tell them how you detect and prevent fraud. Appeal to their moral integrity, educate them about the consequences of fraudulent behavior, and convince them that the likelihood of getting caught is too high to be worth the risk.
Submit
Join our mailing list
Thanks!
Copyright© 2019 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
Contact
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.
Steve Ferrara, Ph.D.
Senior Adviser for Measurement Solutions, AdvancED | Measured Progress

What is the value of deterrent measures to the security of a high-stakes exam?

"My experience in state assessment programs is that deterrence measures are necessary and effective. I used them with, I believe, great effect as the State Director of Student Assessment in Maryland in the 1990s. And I’ve seen them used in considerable numbers of states as a vendor and TAC member."

What are the benefits and/or drawbacks of broadcasting existing test security measures in order to discourage cheating or theft?

"As I see it, the benefits are helping people understand that you’re serious about protecting test security and that you’re demonstrating your commitment to fair and ethical treatment of people involved in testing programs and examinees through transparency. That latter benefit can help counteract the response of some people who may view test security measures and actions as Big-Brother-like. "One disadvantage is that it makes clear to those who are determined to violate test security where to attack; where the vulnerabilities of current measures may be and what measures are not in place."
What is the value of informing examinees of the specific consequences for breaking test security rules?

"Again, transparency is important in demonstrating fairness, deterring bad behavior, and minimizing judgmental errors. And broadcasting consequences is likely to have a deterrent effect on examinees—and don’t forget, on assessment program managers and administrators." Drawing on your experience, what is a practical action that testing programs can implement to deter individuals from cheating or stealing test content?   "I found in Maryland that the most effective deterrent to cheating and stealing was conducting investigations into allegations and other evidence, and imposing fair but harsh penalties when violations were supported by evidence. Over the first two years of our security program, as our records indicated, sloppiness in test security protection and test administrations and outright cheating and stealing disappeared—OK, so maybe some went on underground—and only minor infractions, mostly judgmental errors, were all that we found."
Kim Brunnert, Ph.D.
Director of Psychometrics at HESI Review and Testing, Elsevier Science

What is your opinion of the value of deterrent measures to the security of a high-stakes exam?

"Deterrence is the single most important part of a test security program (but a test security program is, obviously, stronger with all parts: deter, detect, and defend). Detection and defense are important but they are like closing the barn doors after the horse is out. Deterrence is the only part of test security that is preventative. Additionally, by creating a culture of honesty and integrity, programs that use testing can minimize the desire to collude and maximize the validity of the test."

What are the benefits and/or drawbacks of broadcasting existing test security measures in order to discourage cheating or theft?
"The benefits of broadcasting existing test security measures include showing that you are serious about test security and reducing the number of students who will try to collude in these ways. Drawbacks include tipping your hand and encouraging students who are serious about colluding to go into other areas that you might be less equipped to handle or detect."
Sarah L. Toton, Ph.D.
Psychometrician and Senior Data Forensics Scientist, Caveon Test Security
"Given the rate of technological evolution and all the ways that technology can be used to circumvent test security, I think it is more important than ever that prevention is our first priority. Many of the things that we do in test security are reactive. We try to detect who may not have taken the test independently, we investigate examinees, proctors, or test centers, or we implement new security measures to prevent the same thing from happening again. All of this reacting locks us into a cycle of reacting to ever-increasing and evolving security threats. Moving past reacting to thinking ahead and planning for a large array of possibilities is the hallmark of truly effective test security. Possible deterrent strategies include:

1) Make the rules extremely clear and simple to understand for people of all backgrounds. Remind examinees of those rules and the potential consequences on the day of testing, and then enforce them.
2) Encourage a culture of integrity. Have a comprehensive test security plan and tell examinees why this is important. Have a tip line or some other mechanism where examinees can report prohibited behaviors.
3) Use data forensics to identify people who do not appear to be taking the test independently, and do something about it. You can also use data forensics to track your results before and after new measures are implemented and see if there is a decrease in security threats.
"While the three strategies listed above have the potential to be effective test security deterrents, it is important to remember that deterrence is difficult, if not impossible, to achieve unless you publicize what your testing program is doing to ensure that the test is secure. However, one possible risk is that by publicizing your actions, you have just provided more information to the examinees who are trying to violate the security of your program, allowing them to be more effective. I think providing a brief overview is a good idea, because it doesn’t bore or alienate honest examinees, but it also doesn’t provide useful information to examinees seeking to gain an unfair advantage. Another possible downside of publicizing your actions is that if social media or the news media gets involved, the story can be sensationalized and bring a lot of scrutiny to your program. I think both of these can be addressed by having a strong security plan and being very clear and intentional in the messages that you send to stakeholders."
Jarret Dyer
President, National College Testing Association Would you agree that deterrent security measures are becoming increasingly critical for addressing contemporary test security threats?  "Yes, unequivocally. Although I do not believe there is one perfect solution. In many regards, I think this is individual to each exam as it relates to its test security lifecycle. That is to say, depending on how long test security features have been in place, candidates begin to react differently to those deterrents. A newly overhauled security plan produces the intended results for a period of time, however as time passes, test security fatigue can set in. As a practical example, we recently assisted with the administration of a small medical licensure exam. The vendor had reached out for assistance to provide additional logistical support and proctoring during the exam. Test security had always been part of their delivery protocol, but it was time to increase the level of security and change a few protocols. On the day of the exam, proctors arrived wearing clothing that indicated they were professional proctors and conducted the exam with increased security protocols. For example, pockets had never been turned out before this administration, nor had wrists and ankles been examined. At the close of the exam, several of the candidates commented on the increased security protocols and thanked the proctors for implementing them. "This is contrasted, however, with testing eco-systems that have a long history of test security, where candidates have experienced the message of heightened test security for so long that it begins to lack the same impact as it did when they started testing. When test security changes, the impacts can be seen immediately. However, as time marches on, the need to update, reinvent and improve the system continues. In this way, test security needs to be constantly revisited and test administrators and security personnel need to realize that a constant retooling is necessary. Additionally, there is no one-size-fits-all solution. Exam fraud can be equated to the Hydra from Ancient Greek Mythology, a multi-headed beast that attacks from different angles, some of them unseen. In response, test security needs to be multi-faceted as well, striving to both educate and inform, as well as deter and prevent. "The most critical component of a contemporary security measure is the human proctor. As machine learning and artificial intelligence become ever more present in modern test security measures, it remains paramount not to replace the human element from the equation. Proctor intelligence (Pi) remains even more important today as Artificial intelligence (Ai) evolves. In practice, the most secure systems moving forward will not replace one for the other, but will use them in conjunction to augment the Pi with Ai. Recent history is littered with examples of technology attempting to replace, not augment, proctors. A current example is record and review proctoring. While the data suggests that cheating is reduced in this environment, the individuals that do cheat can easily pilfer intellectual property from the examination and distribute it without the ability to prevent this on the part of the owner. This is the perfect example of what was described previously; preventing one, but not addressing all aspects of test fraud." What are the benefits and/or drawbacks of broadcasting existing test security measures and informing examinees of the consequences for breaking those rules? "The benefits outweigh the drawbacks. Clear articulation of the seriousness of the process, as well as an understanding of the importance on the side of the vendor and administrator as well as the proctor, dictate to the candidate that the entire ecosystem takes this seriously. We see the greatest amount of attempted test fraud in situations of ambiguity; when the vendor and administrator or proctor are not on the same page."
Drawing on your experience, what are practical strategies that testing programs can implement to deter individuals from breaking the rules? "What security benefits might a program expect to receive from employing these strategies? Rely on proctor intelligence and train your proctors well. There is no better indicator of candidate malfeasance than the feeling in the pit of a proctor’s stomach when something just isn’t “right”. It is paramount that the vendor fully trusts in the proctor and produces training materials that are well articulated or reasoned. Additionally, communication is key. Candidates need to be informed at least 5 times of the rules and regulations expected of them prior to arriving to take the exam. This not only improves the overall candidate experience but also enforces the seriousness of test security to the candidate. Finally, follow the steps outlined in Proctoring Best Practices (ATP-NCTA, 2015). The guide is straightforward and can be adopted for almost every exam delivery. I’ve been asked for recommendations by many colleagues looking for a solution to align their programs with best practices and this is my go-to for all of them."