The Embrace Change Edition
Rachel R. Watkins Schoenig, Founder & CEO, Cornerstone Strategies, LLC
Ask an Expert:
Rachel R. Watkins Schoenig
Join our mailing list
Copyright© 2018 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.
What are the most important changes the testing industry has undergone in the past 25 years? How have these changes promoted or detracted from test security? 
In broad brush strokes, the same changes that have impacted society generally have impacted the assessment industry as well.  Globalization has had a significant effect on numerous industries, and the assessment industry is no exception.  Over the past 25 years, many programs have seen a significant expansion across jurisdictions as well as an increase in diverse candidates sitting for exams domestically.  The issues that arise with such expansion have created exam security challenges, requiring different protection and investigation considerations as a program undergoes global expansion.  For example, the use of certain types of biometric data may be lawful in one jurisdiction but prohibited in another.  Unfortunately, at the same time programs have expanded, efforts to undermine exam security have expanded as well.  Organized testing rings operate across the globe, taking advantage of the ease of international travel and technology to deliver an unfair advantage to their clients.  As a result, protecting an exam that is delivered in more than one country requires a more sophisticated exam security framework. Technological advances have also had a profound impact on society and significantly changed the assessment industry.  From item writing to administration, scoring and reporting, every facet of the testing lifecycle has benefitted from faster and more efficient computing power and from the use of the internet for sharing information and data.  As a result, we can more quickly develop, administer, and score exams.  From an exam security perspective, technology has provided tools such as biometrics to thwart surrogates, encryption to better secure test content, adaptive testing and alternative item types that make it more difficult to gain pre-knowledge, and additional data to mine for anomalies and concerning trends.  Of course, technology has been a double-edged sword in that it has also enabled the growth of organized cheating rings and also the use of the internet and other digital devices to engage in behaviors that undermine exam security.  Overall, technological advances seem to be a zero-sum game, with the exam security gains and losses balancing out at the end of the day.  What are the most pressing issues for test security today, and how can testing programs – both big and small – prepare to effectively counter these threats?     All programs continue to be challenged with balancing ease of access and customer experience on the one hand, with securing testing assets and ensuring valid results on the other.  For example, continuous exam windows and distance testing provide greater access to tests and help enhance the customer experience.  At the same time however, they increase the risk of exam content or score validity being compromised. Digital devices, the internet, and social media have made stealing and sharing live content or answers easier.  Cost-effectively combating item compromise and pre-knowledge remains a pressing issue for many programs, large or small.  The issue becomes even more complicated due to the continued growth of organized test fraud across the globe. Security works best in layers, so there isn’t one specific tool that will counter exam security threats.  Having said that, there are a number of tools that testing programs can implement, all of which vary in cost and technical difficulty.  On the higher end of the cost-scale, frequent replenishing and rotation of test content can help minimize the impact of these issues.  The use of technology to detect digital devices and data forensics to identify anomalous behavior and response patterns can also be effective.  Trained and observant proctors can also help to counter these threats.  On the lower end of the cost scale, implementing proctor training can be a cost-effective measure to help ensure proctors are aware of what to watch for prior to and during testing.  Reporting tools, sometimes referred to as tiplines, are also relatively inexpensive and can provide an effective tool for honest test takers and others to relay concerns that may be difficult to suss out with data alone.  Finally, simply reminding or nudging test takers early and often of the type of honest test taking behavior to be expected can be a reasonable and cost-effective method for helping to counter these threats.  Taken alone, none of these will be effective.  Taken in layers, however, these capabilities can provide an effective exam security framework for both large and small programs.
"Security works best in layers... There isn't one specific tool that will counter exam security threats."
What is a time in your career when you have taken a leap into the unknown, embraced a new innovation, and experienced a positive change as a result? 

At the time of implementation, the use of facial recognition software during registration for admissions testing was a leap into the unknown.  We were faced with a number of questions concerning the technology as well as the abilities of examinees and test staff to adjust to the new requirements.  I was fortunate to have supportive leadership and implementation teams to help design and implement the new biometric tool.  From an exam security perspective, the leap was definitely worthwhile as the new capabilities resulted in a significant and measurable decrease in impersonation testing attempts.  Equally important, from a change management perspective, the efforts that went into preparing examinees and staff for the change enabled us to incorporate this new security tool in a relatively short time period with minimal disruption to examinees and other key stakeholders.  Technology can be used to steal and cheat, but it can also be used to help better protect tests. What role do you think technology will play in test security in the future? Do you think advancements in technology will offer more benefits to the would-be cheaters and thieves of the world, or do you think new technologies will benefit those of us who are trying to stop them?

Technology itself is agnostic; it can be used for good or bad, depending on the people who create or implement it. For the foreseeable future, I believe technology will continue to be a double-edged sword for traditional testing models, providing new opportunities to cheat or steal content while at the same time providing new opportunities to thwart those efforts.  New types of measurement tools that make use of longitudinal data may provide opportunities to reduce cheating, but I think people will continue to find work-arounds in an effort gain an advantage, regardless of the technology or assessment methods used.
How do you think online proctoring has changed/will change the testing industry? What impact do you see it having on test security both now and in the future? Do you think this impact will be positive or negative?     Online proctoring has provided testing programs the opportunity to broaden access to important assessments while maintaining some degree of security over the identification and administration processes.  Through online proctoring, programs can use visual and auditory data to help secure an exam administration.   For programs with limited access to or use of trained proctors, or for programs using proctors that are susceptible to bribes or pressure to collude, online proctoring can positively benefit exam security.  As technology continues to improve, I anticipate the industry will see more online proctoring capabilities introduced into the market.
"The danger of our industry lies in the extremes: changing so often that we can't have confidence in the results, or changing so little that we become irrelevant."
It often feels that the testing community prefers structure, routine, and consistency over new opportunities for innovation and change, particularly when it comes to embracing and incorporating new technology. What are the benefits and drawbacks of being too reliant on old methodologies versus embracing new innovations?

The data we provide are used to make important educational and career decisions.  As a result, our industry is understandably focused on ensuring that assessments are psychometrically sound and consistently reliable.  The idea of “failing early and often” has limited appeal for testing programs where failure can directly and unfairly impact an individual’s educational and career opportunities, and even lead to potential legal liability for the testing program.  Thus, embracing new innovations without appropriate supporting data and effective implementation procedures has real-world consequences that may not be easily remedied.  Faced with this reality, the industry is understandably concerned about consistency and reliability.  Taken to an extreme, however, a dogged adherence to structure and routine can result in a corporate (and even an industry) culture that fails to appropriately explore new methods or innovations and ultimately renders itself obsolete.  The danger for our industry lies in the extremes:  changing so often that we can’t have confidence in the results, or changing so little that we become irrelevant.   In light of this, I think it’s reasonable to suggest that, while our industry may not be blazing new technology trails, we should be “fast followers”, highly attuned to what is happening with new software and hardware capabilities and adopting those when sufficient evidence exists to ensure psychometric validity.  Keeping a pulse on new capabilities will better position programs and the industry to incorporate emerging methodologies or technologies in a timely way while still producing reliable and meaningful results.    What advice would you give to leaders in this industry who have the desire to embrace innovation, but can’t take that final step to actually implement it?     I would advise not to be afraid of change; rather, plan for it, including helping your organization positively respond to change.  I think there’s a natural tendency to want to maintain the status quo.  It takes wisdom, courage, and fortitude to change – both as a leader and as an organization.  For leaders, identifying innovations that can meaningfully improve measurement, reliability, speed, access, or security is just one step of a longer journey.  Leaders still need to understand and balance whether the costs of change are worth implementation.  Implementing new innovation always comes with both hard and soft costs.  As simple as it sounds, it’s important to weigh both the hard and soft costs of any new innovation prior to taking the leap of implementation.  Is the benefit to examinees and/or the organization clearly articulated?  Is the risk of not embracing the innovation significant enough to outweigh the cost of changing?  If the balancing of costs and benefits is weighted to the benefits side, then the next step is to help your organization and other stakeholders embrace the change. Moving to implementation is more than just making the decision to change.  As a leader, helping your organization be open to and capable of embracing change is important.  That can start with clearly articulating why a new innovation is needed, which can help push through the inertia inherent in an established process or system.  Leaders who help plan for change and embrace it as a positive can help internal and external stakeholders buy-in to the change.  I mentioned above one of the technology changes I have helped lead was the introduction of facial recognition in the admissions industry.  We didn’t just “flip the switch” to implement photo upload and facial recognition at registration, however.  We articulated a clear “why” as to the reason for the change and involved individuals throughout the organization as well as proctors, teachers, and test takers in the change effort.  We engaged in an agile development process with frequent prototyping at each step.  We regularly solicited feedback from stakeholders and test takers and messaged the change on a frequent basis before ultimately going live.  Doing so helped to ensure a smooth transition from a technical standpoint as well as buy-in from an internal and external stakeholder perspective.  Once implemented, the organization experienced a dramatic and measurable decrease in impersonation testing.  Leaders who are looking to take the leap from desire to implementation should plan for both the technology changes as well as the people and process changes that need to occur in order to fully leverage new innovations.
"I would advise not to be afraid of change; rather, plan on it."
What are some of the most exciting new test security innovations you have seen or heard about recently?   This is an exciting time to be in testing, as there are a number of new innovations or technology applications emerging.  For example, ACTNext is working on merging education and assessment in some exciting new ways that may offer additional security benefits.  Advancements in automatic item generation, data forensics, and artificial intelligence are also providing new tools for securing test content and identifying concerning behavior.  Blockchain and new biometrics are providing additional authentication and identification capabilities.  Gamification, virtual reality, and mixed reality also offer new opportunities to not only measure skills, including soft skills, but offer unique exam security benefits as well.   Looking forward 10 years, how do you think this industry will change?  In the education and training space, I think we’ll continue to see a merger of education and assessment activities as well as the use of longitudinal data to measure skills and deliver secure results.  I also anticipate efforts to scrape data from programs that people already engage with on a regular basis.  For example, the video game Fortnite has millions of concurrent users who can play the game in teams.  Theoretically, if given permission to do so, a testing organization could tap into the data being generated during the games to determine whether it can be used to measure soft skills such as teamwork or conscientiousness.  Artificial intelligence, machine learning, and deep learning will impact the industry in ways that I think we have only begun to imagine, both from a measurement perspective and security perspective.  At the same time, artificial intelligence, machine learning, and deep learning also pose new ethical and privacy issues.  I anticipate the industry taking a more active role in discussing the ethical use of data, exploring the need to understand testing constructs being measured, and refining the ongoing obligation to understand and eliminate potential bias in results. 
Use the latest scientific research conducted by Caveon to evaluate all of your options and make the best decisions for your testing program.
Read more →
You might also like...
More Reads
Celebrating innovation in every corner of our industry, this column highlights companies, organizations, and individuals who are pursuing dynamic testing transformations.
Read more →