The Prevention Lockbox
Ellen_Julian
Ellen Julian, Ph.D., Principal & Founder, Julian Psychometric Consulting
calendar-icon_whitecircle
6 Practical Ways to
Prevent Cheating
Dr. Ellen Julian became fascinated by tests in the second grade and has been trying to figure them out ever since. Trained as a psychometrician, she graduated from crunching the numbers to running testing programs and now leads Julian Consulting, helping testing programs grow and make good decisions. Ellen was the 2018 President of the Performance Testing Council and hosts the DC-area Testing and Certification Salons.
You might also like...
More Reads
Ever wonder if SmartItems actually deliver on their promises? SailPoint testifies that when used on two of their certification exams, SmartItems provided benefits such as stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation.
Celebrating innovation in every corner of our industry, this column highlights companies, organizations, and individuals who are pursuing dynamic testing transformations.
Submit
Join our mailing list
Thanks!
Copyright© 2019 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
Contact
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.
Detecting cheating as or after it happens requires vigilant proctors and fancy statistical analyses, not to mention a bevy of lawyers if you want to take action. However, putting systems in place to make cheating more difficult may make those expenses less necessary, and your world more peaceful. Here are some steps a testing program can take to reduce opportunities/inclination to cheat:
1. Repeaters may have an advantage from seeing the same items repeatedly. Have at least one of these mitigators (preferably all three) in place:
  • Multiple forms of the test: Consider building and publishing forms two-at-a-time and leaving them in the field twice as long. With multiple forms in the field, repeaters take a different form, increasing fairness. Since form publication is labor-intensive and where many errors happen, doing it less often can help increase quality and efficiency. Make sure your test administration vendor will assign the unseen (or less-recently-seen) form of the test to repeaters.
  • A wait-period between retakes: Where employment is dependent on passing a test, waiting periods can make failing a test even more traumatic. One option is to allow one immediate retest (if a second form is available) and then require some time for remediation before a third attempt.
  • A limit on retakes (annual or lifetime): This is especially important if competence requires more than a test can cover. As time since education increases, the other skills may fade. Sometimes it’s a mercy to stop people from throwing their money away; other times it would be cruel to crush their hopes. This is a tricky one.
2. Even if you have only one form, you can make it seem like many by scrambling item and response-option orders.
  • Most item-banking/test-publishing systems offer scrambling as an option. Many offer the ability to publish a pool for random selection of pretest items.
  • Be careful of the added complications, like scoring, QC’ing answer keys, tracking examinees’ item challenges, and storing item-response strings. Scrambling items is one level of complication; scrambling options adds another.
  • This may increase measurement error over what’s reported because of any impact of item or option placement on difficulty, but the impact on any individual can be assumed to “average out.”
  • This option is totally worth the extra trouble! Start with scrambling item-presentation order.
3. Use item templates, where items may look the same except for critical details.
  • Don’t add memorable details, like names, into items. This helps avoid sensitivity issues, too.
  • Conversely, if you include colorful details in items, they may be all you need to change to make it a new item. Don't assume the difficulty won't change, but this could reduce the item writing and reviewing load.
  • Implementing the item-template concept doesn’t have to be complicated. It’s easiest to start with computational items—just change the numbers. Again, don’t assume that the difficulties will be the same.
  • Full implementation takes some set-up, but can reduce item-development costs later. Look at the work done by Gierl and the ABIM on Automated Item Generation.
4. Use testing windows (with delayed score reports) instead of continuous testing.
  • Opening the test for only a short period of time (e.g., a one-month "window") lets you collect and analyze the data before reporting any scores.
  • This allows you to score items on their first use—pretesting is no longer necessary—so new forms are easier to build. If new items aren't performing well, don't score them. if they have good statistics, include them! A healthy anchor test of previously-used high-quality items will allow you to equate your existing passing standard to this new form.
  • Cheaters must be quicker and more organized because the forms are available for shorter periods and go away before scores are reported.
  • You can identify and correct problems before issuing any scores—it’s much easier to decide not to score an item (or a person!) before even preliminary scores are issued.
  • Using testing windows avoids some retester-frequency issues—by the time someone knows they will need to retest, the form they took is no longer available. They must wait until the next window to test, so there’s an enforced waiting period.
5. Use pretest-item pools, where each person’s pretest items are selected at random from a larger pool of pretest items published at the same time as the test.
  • Each examinee sees a random selection of pretest questions, making it seem like there are lots of different test forms.
  • Ideally, the form is pulled from the field as soon as the last item is seen by its final examinee. The optimal size of the pretest pool is easy to calculate using the number of examinees who will see the form (N), the number of pretest items each examinee sees (I), and the minimum number of exposures your psychometrician has requested for each item (Min) determine: (N*I)/Min.
    • For example, 500 expected examinees * 40 pretest items per examinee / 250 desired responses per item = a maximum of 80 items in the pretest pool. The smallest possible pool has enough items to build a complete test, in this case, 40.
  • Using pretest-item pools also helps reduce the number of test publications, which are expensive and error-prone, and makes sure that as many items as possible are getting pretested.
6. Use behavioral nudges to support and inspire examinees to do the right thing.
  • Remind examinees at the critical moments—as they register and then again as they start their test—of their desire to be good people and to do the right thing.
  • Drop the legalese: Use simple, plain language to describe rules and consequences, to provide clear guidance about what rule-following looks like, to explain why it’s important, and to remind them of the consequences of cheating, for both themselves and society (e.g., incompetent practitioners).
  • Indicate that others are taking the test honestly, too. Focus on the positive side, the people who are doing it right.
  • Images of watching eyes can help people be honest. Seriously.