Dr. David Foster, CEO of Caveon Test Security
March Edition
calendar-icon_whitecircle
Dr. David Foster
Data-Driven Success:
Use the latest research to make the best decisions for your testing program
The Difficulty of Item Types: DOMC vs Multiple Choice
In 2017, Caveon ran a series of studies that compared the traditional multiple-choice (MC) item with the Discrete Option Multiple-Choice (DOMC) item. The studies were true experiments, comparing the MC item with the DOMC item on several properties. A white paper about DOMC can be found by clicking the document at the bottom of this page. A common question asked about MC and DOMC items is, “Which is more difficult?” An experiment using tests made of 20 questions about Harry Potter, answered by over 2,500 fans, confirmed that DOMC questions are more difficult, in this case, by about 4%. The question then is "why". Why is DOMC more difficult, even if it just by 4%?
The question then is “why.” Why is DOMC more difficult, even if it is just by 4%?
Experiments by researchers at the University of Dusseldorf, which used questions from medical education courses, showed that the difference in difficulty is likely due to the fact that MC questions provide more test taking cues than DOMC questions. You can view that  paper here. To illustrate the use of test-taking cues, here is an example from one of Caveon’s Harry Potter questions. Looking at it, one can see how this phenomenon might explain difficulty differences:
This is a fairly difficult Harry Potter question, even for fans of the show or books. Only 43% of the 2,500 test takers knew that the correct answer was 5. For those who don’t know the answer and who need to resort to guessing, the chance of getting the question correct is .20 or 1/5.
However, because this is a traditional multiple choice question and all of the options are shown to the test taker at once, a test taker who is trying to guess will use every bit of information available to try and increase the odds of getting it right. In this case, because the correct option is not one of the end values of the set, 4, 5, 6, 7 and 8, a test taker who has honed their test taking skills will eliminate the two end numbers (4 and 8) and will guess one of the “internal” numbers (5, 6 or 7). In doing so, they increase the chance of getting this item correct from .2 to .33. 
Now, let’s compare that to a DOMC item.
As you can see, DOMC does not allow this type of test taking strategy because the options are presented one at a time, and in a random order. Test takers cannot compare the options and determine which of the options are “internal” and which are “ends” of a sequence. They cannot eliminate options in order to improve their chances of guessing. While there may be other contributors, the ability to use test taking cues in MC questions seems to reasonably explain the difference in difficulty between MC and DOMC. Though the DOMC difficulty higher is lower than MC, it eliminates test taking skills and can be characterized as providing a more pure measure of a person’s knowledge or skill level.
You might also like...
More Reads
Test security expert Jim Wollack speaks about the value of test security, how it has changed in the past decade, and what he thinks the future holds.
Read more →
Ask an Expert: Jim Wollack
To learn more about the DOMC item type, read this article by David Foster:
toc_icon_BLUEcaveon
Home
Featured Articles
→ Protection, The Primary Purpose of Security
→ It's a Risky Business
→ Jim Wollack Speaks
→ 5 Steps to K-12 Test Security Success
Short Reads
→ Interview: John Fremer
→ Industry insider: 30 Questions for Walt Drane
→ Interview: John Fremer
Contact
caveontoc_icon_BLUE_desktop
Submit
Join our mailing list
Copyright© 2018 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
Contact
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.