Protection: 
The Purpose of Security
Dr. David Foster, CEO of Caveon Test Security
March Edition
calendar-icon_whitecircle
Dr. David Foster
Eventually (after several unexpected twists and turns) I started Caveon Test Security, the first company of its kind in this industry. I have now spent nearly 15 years working in the exciting (to me and other converts) and often-overlooked world of exam security. Even though I have been promoting the value of test security for nearly two decades, it wasn’t until just a few years ago that I truly began to understand the meaning of the term “security” and why our industry needs to care about it. Oddly enough, my new understanding began by looking at how security is approached in areas other than testing. My quest took me through a wide variety of industries including military base security, home security, casino security, financial security and an area that currently dominates news headlines, the security of information systems that protect our personal information stored on computers and the cloud. In learning about how these other industries approach security, I soon came to realize that even though these security efforts are vastly different in many ways, they all have core principles in common. In fact, the basic principle of security for each is the concept of protection — they have something valuable, an asset of some sort, that needs to be protected from those who would try and undermine it. I learned that this concept is the starting point of all effective security, whether in the field of testing or elsewhere. Shame on me for taking so long to figure it out. Protection is a goal, not a method or tool. All industries that have valuable assets—and we can include our own industry on that list—should have the the same goal of protecting them. If those of us who work in the field of testing accept that we offer something valuable to this world, something that needs to be protected from people who want to tarnish it, it should become our #1 priority to protect it. To do so, we must ask ourselves a few central questions:
What are we protecting? What are we protecting it from? And what are the best ways to protect it?
What are we protecting?
The first question we need to ask ourselves is: what are we protecting? Logically, if you don’t know what you are protecting, you will not be able to protect it. In the field of testing, what is our most important asset? Is it our exams and items? Is it our reputation as specific programs? Is it the testing industry as a whole that we are trying to protect? This might come as a shock to you, but while these are significant assets, they are not the most important. Based on a review of the Standards of Educational and Psychological Testing, Fourth Edition, published in 2014, our most important assets are useful test scores.

An exam is only valuable if the test scores it produces can be confidently used for their intended purpose. Unfortunately, right now, there is a persistent doubt that surrounds the validity and usefulness of test results. Those who use test scores, as well as the general public, question whether test results are of sufficient quality, whether they have been compromised by test fraud, or whether they are fair for all test takers. With the number of cheating scandals reported every month in the media, it is hard to blame people for having this doubt. This is what we need to protect – the validity of our test scores. Without it, our exams become useless.
What are we protecting against?
With our focus now directed at protecting useful test scores, we need to look at the second question: what are we protecting test scores against? Another way to think about this is to ask ourselves, “What are the threats to useful test scores, and who/what is behind those threats?” Thankfully, the list of threats is relatively small, and others have already done the heavy lifting and identified them for you. (Foster and Miller, 2012; ATP, 2013; ITC, 2014). The good news is that now that we have identified the threats, we can take steps to prevent them, and once we prevent them, we have successfully protected the test scores, and they can be confidently used!1 Even when threats are eliminated, the question of who is behind the threat might never be clearly answered. Many of these nefarious individuals or organizations are never uncovered (after all, they are actively working hard to not get caught.) While disappointing, the goal is to stop the threats and stop the individuals behind them. Catching those individuals and bringing them to justice is a bonus. A guilty pleasure of mine is knowing how much frustration and anger my security procedures are causing people who want to cheat on tests or steal exam content.
Rating and Ranking Threats
On a practical note, it is important to remember that while there are only a few categories of threats, our security resources are always limited. Budgets are a very real factor for anyone working in test security. The good news is that if funds are allocated smartly, limited budgets need not stop us from protecting our scores. For this reason, the first step of any security efforts should be to narrow the list of potential threats to those that are the most likely and would be the most damaging. This will be different for every single testing program. For example, for one program the most dangerous threat may be having the questions stolen and shared worldwide on the Internet; for another it might be the threat of proxy test takers. There is no one-size-fits-all answer to what the worst threats are, and each program must carefully analyze and determine this for themselves. Fortunately, determining which threats are most important to your program is easy! A quick and relatively painless risk assessment analysis (which consists of rating and ranking the threats) will do the trick. I would suggest reading more about this process in Jamie Mulkey’s article in this e-zine, or by accessing a do-it-yourself risk assessment for free at Caveon’s website.
How do we stop them?
The final question remains: how do we stop these threats? How do we quickly find a breach that has just begun? How do we inhibit, discourage, deter, and even scare the individuals behind the most likely threats? The methods or protection (what I call “security solutions”) fall into three categories: Prevention, Deterrence and Detection/Reaction. Each category is equally important. However, they work best when used together, with each one complimenting the other's protective strengths. Let me talk a bit about each category to illustrate its usefulness. Prevention. If you completely prevent all threats, then, problem solved! Woohoo! Unfortunately, while I dream of a world in which all threats can be completely prevented, it probably isn’t possible. We can, however, prevent some threats completely, and some threats partially. Best practices, combined with common sense and creativity, will help you to build and implement preventive solutions.
Prevention, as the word implies, makes it impossible for a threat to work in a particular situation. Let me show you an example from the real world. Figure 1 shows a picture of a bollard, a physical structure designed to control vehicle access to public areas, preventing the possibility of a car being used to attack civilians.
As for an example from testing, a computerized adaptive test (CAT) does not display all of the test questions in the question pool. A test taker attempting to harvest the questions will only be able to steal the questions they see — a small percentage of the total item pool. Simply by using this test design, you have prevented a thief from stealing all your questions. Another example from testing: cheating by “copying” off someone else’s test has been effectively prevented with today’s use of computerized exams where start times are naturally offset, and questions are administered in a random order. Deterrence. Deterrence is a psychological prevention strategy that targets the motivation of individuals who are considering cheating on a test or stealing questions. To deter someone from unwanted behavior, you put in place procedures that make a person not want to carry out the threat. Your procedures could convince them that they will get caught, or that what they are planning simply won’t work, or that it simply isn’t worth the risk. For the average test takers, simply informing them of the security rules and outlining the consequences of cheating is deterrence enough. For those more determined fraudsters, you will need a stronger set of deterrents.
A security guard sign is a form of deterrence against trespassers. The sign does not prevent a person from trespassing, it merely tries to convince them that trespassing would be unwise. While the testing industry also makes use of signage (such as “no cell phone signs” in a testing room), we can also deter fraud in many other ways. One example is when the Graduate Management Admissions council publicized a successful takedown of a website/company that had stolen and sold GMAT questions. Descriptions of fines, penalties, seizures, legal sanctions, and financial loss were published and distributed as a deterrence to all other organizations and individuals that might try and do the same thing. One thing to remember is that deterrence requires purposeful communication with the individuals behind a threat. Keeping deterrent strategies quiet will not work; you must convince would-be wrongdoers that procedures are in place that will catch them, and that they will surely be punished. Detection/Reaction. The third protection solution is detection/reaction. This is a two-step process. First, a set of detection measures are put in place, usually with the goal of detecting the initial attack from a threat. Second, after an attack has been detected, a series of pre-planned and associated reactions are triggered to deal with the attack and limit its consequences. Figure 3 shows how the collision detection system for automobiles works. Sensors detect an object in front of the car and warn the drivers. If the drivers don’t react in time, the car can stop itself by applying the brakes independent of the operator. The sensors detect a possible collision, and then react with warnings and positive action.
One example of a detection/reaction solution in the testing industry is a vigilant proctor. The proctor should be well-trained in how to detect someone cheating on a test, but also in how to react and deal with the cheater. These two steps always work in tandem, because it doesn’t make any sense to detect a breach without having a pre-planned response to deal with it. I also like the term “reaction” because it suggests an almost automatic response to a detected breach. Another example of a detection solution is a data forensic analysis of test results. Unusual patterns of responding, caused by various types of cheating or by theft, are often apparent only by statistical analysis. If the analysis occurs early enough, it can allow a testing program to respond quickly and avoid extensive damage. Repeating the Protection Process. Threats change, with new ones on the horizon each year. Imagine a national defense strategy that fails to acknowledge ever-changing threats such as cyber hacking and transnational-terrorism. It would be alarming and ineffective. The same is true for our test security process. The testing community needs to acknowledge that in order to protect our most valuable assets — our exam scores — we must continually evaluate current threats, as well as our process for protecting against them. Complacency and protection do not mix. Protection is a dynamic process, one that requires innovative and exciting solutions. To succeed, we must continually ask ourselves these three questions: (1) What are we protecting? (2) What are we protecting it from? and (3) What are the best ways to protect it? As the world changes, so will the answers. It is only through this constant evaluation that we can be confident that our solutions are working, and that we are truly reaching the ultimate goal… protecting the usefulness of our test scores.
“Protection is a goal, not a method or tool. All industries that have valuable assets—and we can include our own industry on that list—should have the goal of protecting them.”
I’ve been involved in the field of testing since 1982. I started my career as a testing program manager and psychometrician, where I experienced first-hand the use of test security procedures.
If we can answer these questions, well, the battle is pretty much won. You might ask, “Dave, isn’t protecting our exams what we are doing today, and what we have been doing for the last century?” I have asked myself that question many times, and it is after many hours of consideration that I have come up with an answer. Here it is: “Not really.” What we do today is simply accept security practices that have been in place for decades, even centuries, without considering if those practices still work, still protect our exams. In all honesty, the persistently large (and even growing) number of security incidents seem to suggest that perhaps the security measures we have relied on up until now don’t really work anymore. Maybe my instinct is wrong and our current security measures are the best we can do, but we need to look closely at them to find out! We in the testing industry work hard to create exams that produce valid test scores, and we need to at least consider that we are putting our own tests at risk by not upgrading our security policies and procedures. We work too hard to just throw it all out the window on the assumption that we are doing everything in our power to keep our exams safe. The first step is to answer the three questions listed above, for as the various security industries listed above know, the continual evaluation of security practices is the first and most fundamental step towards protection. Let’s get to it.
References
  1. Association of Test Publishers (2013). Assessment Security Options: Considerations by Delivery Channel and Assessment Model, produced by the ATP Security Committee and found at: testpublishers.org
  2. Foster, D. F., & Miller, H. L., Jr. (2012). Global Test Security Issues and Ethical Challenges. In M. M. Leach, M. J. Stevens, G. Lindsay, A Ferrero, & Y. Korkut (Eds.), The Oxford Handbook of International Psychological Ethics, Oxford University Press, Oxford.
  3. International Test Commission (2014). The ITC Guidelines on the Security of Tests, Examinations, and Other Assessments, found at: intestcom.org
Share:
You might also like...
More Reads
Test security expert Jim Wollack speaks about the value of test security, how it has changed in the past decade, and what he thinks the future holds.
Ask an Expert: Jim Wollack
The topic of this month's advice column: "Help! I'm already taking precautions like web patrol. Is there something I can do to prevent theft from happening in the first place?
Ask Agatha
toc_icon_BLUEcaveon
Home
home_icon
Featured Articles
star
→ Protection, The Primary Purpose of Security
→ It's a Risky Business
→ Jim Wollack Speaks
→ 5 Steps to K-12 Test Security Success
Short Reads
bookicon2
→ Interview: John Fremer
→ Industry insider: 30 Questions for Walt Drane
→ Interview: John Fremer
Contact
envelope_white
cross_closecaveon
caveontoc_icon_BLUE_desktop
Contact
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.
Submit
Join our mailing list
Copyright© 2018 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use