As I’ve thought about the many innovations that have impacted my life (cell phones, medical advances, and transportation, to name a few), I can’t help but turn my attention to my calling of 36 years – testing. Within the testing industry, I have selected seven incredible innovations that I think have had the biggest impact on this field. I've included a brief description, context, and robust reasoning on why each innovation is one of the greatest of our time. (Note: I will mention if I have personal experience with one of these innovations.)

The MC Question

The multiple-choice question was invented in the early 1900’s by Frederick Kelly. He created it to standardize the testing process, reducing teacher scoring errors and bias. The item was first used on the Kansas Silent Reading Test in 1915, but was quickly followed two years later by the wholesale adoption of the format for the Army Alpha test. The popularity of the format continued and is apparent to all, even those giving or taking tests today. Few innovations can boast such a long life, virtually untouched by aging. I can attest to significant involvement in both taking and giving multiple-choice questions throughout my life and my career. Figure 1 shows an example of the format from the Kansas Silent Reading Test:

Answer Sheets & Scanners

The first Optical Mark Recognition (OMR) scanner was created in 1932, followed by 3 decades of many newer inventions and patents. Today, scanners continue to be used in K-12 and higher education where paper-based tests are given in large quantities. Figure 2 shows a typical OMR sheet and a scanner from the end of the last century. The OMR technologies from that time significantly advanced and solidified the use and popularity of paper-based multiple-choice testing. The OMR equipment allowed the quick collection and scoring of student capabilities, particularly of end-of-year, summative tests.

IRT & Adaptive Testing

Item response theory, or IRT, was introduced around 1950 by Frederic Lord, George Rasch, and others. Compared to classical test theory, IRT is generally believed to have brought greater sophistication to test analyses, greater ability to demonstration reliability, and new applications, such as computerized adaptive testing (CAT). CAT allowed tests to be administered more quickly (using fewer items) and more securely. In perhaps the first worldwide use of CAT for a high-stakes test, at Novell, a software company certifying the competence of its administrators and engineers, I used IRT analyses and CAT in over one million exams administered globally from 1990 to 1997. Figure 3 shows a typical IRT-based item information curve.

Computerized Testing

Some people equate computerized testing with CAT, but computerized testing is a much greater concept. Certainly, moving testing in the 1980’s from the limitations of paper booklets and answer sheets took CAT from the drawing board and into reality. But it brought so much more, even to tests that were not adaptive. With tests being computerized, scoring was quicker, even instantaneous, as were the decisions based on those scores. Accommodations for disabilities could be made on-the-fly, such as text-to-speech and larger fonts. Such testing also encouraged new item formats, such as dragging and dropping objects, the use of speech, scoring flexibility, the use of multiple languages in tests, having people complete tasks during the exam, taking the tests at home, and many, many more. The first computers to provide this flexibility were mainframes controlling so-called “dumb” terminals. Then, networked personal computers grew to popularity in schools and homes. Today, such computers allow tests to be given on cell phones and tablets and other mobile devices. This evolution in computer technology can be seen in Figure 4.

The Internet

In the 1990’s, I had heard of the new technology referred to as the “World Wide Web” or Internet. While I didn’t understand how it worked, nor was I able to predict its usefulness in the field of testing, one event in the mid-1990’s showed its clear value. At that time, Novell was spending many thousands of dollars to collect testing results from far-flung reaches of the world. In one such event, it cost over one thousand dollars in long-distance charges from just one weekend of using a modem to collect data from a test site in Poland. Then, almost overnight, we used that new thing, the Internet, to complete the same outcome for no cost at all! Of course, today’s tests can be given using the Internet (properly called “online testing”) where testing data can be stored, analyzed, and much more. The Internet has permeated our field just as it has done elsewhere. Figure 5 presents a conceptual view of the Internet today.

Discrete Option Multiple Choice

As an avid user and firm believer in using technology in testing, I invented a new version of the multiple-choice question where all of the answer options aren’t revealed initially, but presented one-at-a-time until the test taker responds either correctly or incorrectly. A colleague of mine called this the Discrete Option Multiple Choice (DOMC) in 2009. A sample DOMC item from www.trydomc.com can be seen in Figure 6. The DOMC format makes the use of testwiseness (the skill of using testing cues to one’s advantage) from multiple-choice questions almost impossible, thereby removing a 5%-10% blight on test scores – scores that the testing industry seemed to have accepted over the decades as a necessary evil. DOMC also reduces security problems and promotes fairness. It has significant support from scientific research and is used successfully in a number of operational testing programs today.

The Caveon SmartItem™

In 2018, Caveon introduced the SmartItem. This is probably the most important innovation in this list. SmartItems are defined by three main characteristics: (1) the SmartItem covers the complete breadth and depth of a defined skill, (2) the SmartItem is viewed as different by each test taker, and (3) SmartItems vary randomly for test takers. Supported in its use by psychometric theory, it is not unusual for a single SmartItem to vary in hundreds of thousands – or even millions – of ways, including in difficulty. SmartItems are amazing in that they completely eliminate threats from item thieves and harvesters, along with protecting against almost all forms of cheating. Some other advantages include saving costs on test development and test administration. To test takers, SmartItems appear as any other item. Any item type can be converted to a SmartItem. How have the above innovations impacted your testing experiences? Whether from a student’s position, career capacity, or otherwise, if you have any thoughts on the above innovations – or if you can think of any other innovations we may have missed – we’d love to hear from you. Email us at info@caveon.com