The Embrace Change Edition
Dr. David Foster
Dr. David Foster, President & CEO, Caveon
The Eight Barriers to Change
Why don’t more people embrace creative approaches to testing? David Foster has given this question some careful consideration throughout his career. Here are some of the possibilities he came up with...
“We need to stick with the standards.”
Some people confuse traditional long-standing methods with standards. They would argue that as a testing standard, traditional multiple-choice questions cannot and should not ever change. Of course, multiple-choice testing is a method, not a standard.

“I’m good at what I do.”
People are comfortable doing what they have learned to do and what they have always been doing. Change is difficult.

“How do I know it will work?” Some people are afraid of what will happen if they try new things. They are unsure whether a new approach will bring the promised benefits, so they opt to play it safe and avoid the new risks.

4. “It needs more research.” Some people rely on the solid – but fairly slow process – of scientific research.

5. “Who else is doing it?” Some people are late adopters, willing to let others blaze the trail. They do not want to try new methods without being confident that it has already worked for others in the industry.

6. “I don’t have the time.” A person may not have the time to learn about and understand the value of new tools and procedures.

7. “That’s hard to do.” Using and implementing new technology can take substantial time and resources, particularly at the beginning. Some individuals may not be willing to undertake the initial work it requires to implement new ideas.

8. “It scares me a bit.”
It’s true that developing and using new processes or tools may not immediately result in a positive experience. That’s the risk one takes with innovation. For some, it is better to avoid that risk altogether.
I have used the title of Psychometrician often during my career, but it never seemed to fit well. I think a better title for me would be “Psychometric Technologist”. The title describes a person devoted to finding innovative ways to support the Standards we all follow in testing. Regarding the value of innovating, all I can really offer is my testimonial based on my personal experiences. Using technology, even technology that has never been tried before, can have amazingly positive effects. During my career, I have had the opportunity to encounter some significant testing challenges, roadblocks, and problems. There must be 20-30 clear examples of this over my 36-year career. Let me describe a few of them. In about 1993 when I was creating certification tests for Novell, a prominent software company at the time, we were getting hammered by our candidates and partners who complained that our tests were trivial, measuring textbook learning instead of real-world experience. Of course, the criticisms were mostly valid. It was clear that instead of asking about “how to do something” using Novell software, it would be better to ask the candidate to actually complete the tasks in the software. I received the go-ahead to create a simulation of NetWare, Novell’s flagship networking software. The simulation interacted with the test administration system and with our test development system so that it was easy to create items that asked the candidate to complete tasks. Scoring was straightforward, and the exam took no longer than the previous version. We still used multiple-choice questions where appropriate to measure other skills. To make a long story short, after implementing the new exam, the complaints about Novell’s exams virtually disappeared overnight. The simulation cost about $250,000 to build and was worth every penny. It was used in many certification exams, practice exams, and quizzes following training courses. Later psychometric analyses revealed that the items performed better than we had hoped. Follow-up validity studies confirmed that certified candidates were well prepared to work for Novell clients and partners. The use of innovative technology was motivated by a problem, and at the time there was no research or experience available to help implement it. My only choice to solve this dilemma was to take a risk and move forward.
In a
blog post
a few years back, I discussed how technology can solve some of the testing industry’s problems and suggested that as an industry, we haven’t yet introduced technology in a creative, effective way. The problems of the past still exist, and a personal review of a large sample of computerized exams built today reveals that 99% of them are simply paper-and-pencil tests that are administered on computers. We are living in the past, and it’s not doing us any favors. Why don’t more people embrace creative approaches to testing? Why don’t we have more adaptive tests, an innovative method that has been around for over 30 years? Why are the vast majority of items used in tests still traditional multiple choice with four options and one correct answer? Why aren’t scores provided immediately after most tests? Why aren’t essays scored automatically and quickly instead of using raters? Why aren’t more tests given on-demand instead of within a few days every year? Why do we travel to a location to take a test rather than take it in our own home or office? Why does it take weeks or months to fix a compromised or broken exam? Why are we still suffering a growing number of security breaches? I have given these questions careful consideration over the years, searching for answers. Here are some of the possibilities I have come up with (an incomplete list, I’m sure):
You might also like...
More Reads
Read this month's Caveon Bulletin to learn about recent publications, upcoming conferences and webinars, and for exclusive access to past webinars & events.
Read more →
Get to know your colleagues in the testing industry a little bit better as they answer fun and intriguing questions from Marcel Proust's 18th-century parlor game
Read more →
There are certainly many reasons not to innovate or use others’ innovations. But there is little to be gained from following the status quo, and much to be gained from abandoning it. We still have major problems to deal with, and current and yet-to-be-discovered technology solutions can provide the welcomed light at the end of the tunnel.
Here is a solution for a simpler problem. At around the same time, Novell candidates experienced some difficulty with the translation of specific words in exams. It seems some technical words that are usually well known in English (e.g., Spooler) weren’t easily translated. While the translated word was in the candidates’ native language, it was not recognized or understood by many. Since some candidates did understand the translated term, we could not simply revert to the English word. Our general solution was to provide the current translated version of the item, and included a button on the screen labeled “English.” When pressed, this button would show the item in the original English version. Including the “original language” of the item erased any confusion and allowed candidates to complete the item without the negative influence of that particular problem. The dual-language item was new and made possible by the computer administration of the certification exams. Later psychometric analyses indicated an improvement in item statistics coincidental with the change. That one technology innovation caused a major complaint about our translated exams to go away forever.
"There is little to be gained from following the status quo, and much to be gained from abandoning it."
Join our mailing list
Copyright© 2018 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.