Aimée Rhodes, CFA Institute
April Edition
calendar-icon_whitecircle
imageedit_16_5083233281
Ask an Expert:
Aimée Hobby Rhodes
Aimée Hobby Rhodes is the Director of Exam Security at CFA Institute. Ms. Rhodes is responsible to ensure that CFA exams are administered securely by evaluating strategies and developing innovative security management techniques. She and her team review policies and procedures with an eye toward security, oversee web monitoring, analyze data forensics, performance statistics, and unusual behavioral indicators to identify threats to the CFA program. They also evaluate cheating technologies to determine how they can be employed in the testing room, develop and test pilot programs to address security concerns, and provide exam supervisors and proctors with materials and methods to identify and report unusual behavior in the testing room.
Do you think it is important for the field of test security to embrace innovation? Why?
"I think it’s essential for test security professionals to embrace innovation. In today’s super competitive world, we know test takers are looking for any advantage to get ahead. We hope most of their energy is put toward studying for our exams and expanding their knowledge, unfortunately, we know some test takers turn to unethical means to get ahead. Many of those means involve innovative ideas. In order for test security professionals to protect the integrity of our exams, we need to be aware of what are test takers are doing and how they might take advantage of an unfair opportunity. Staying on top of and even ahead of this kind of innovation is crucial."
In the past decade, what have been the most important innovations in test security?
"The past decade has seen many advances in test security. The regular use of data forensics to identify cheating and item harvesting is one great example of that. The use of data analytics to identify threats to an exam before that exam is given is another."
Data forensics is increasingly seen as one of the core test security measures that a testing program should use to protect their exams. What are some of the innovations in data forensics that you find particularly exciting, and what are some innovations you would like to see in the coming years?
"I find all digital keystroke analyses very interesting. Item response time analysis and response time stamps, answer change data, and skewed timing are also exciting to me. This information provides an investigator a great deal of information about how a test taker completed an exam. It’s almost as if they lay a path of breadcrumbs as they worked their way through the exam. This comprehensive picture of exam day can absolutely help determine whether or not a test taker was acting ethically. In the coming years, it would be great to see these analyses automated and used in real time so they are able to flag problematic candidates as they test. This is an innovation I look forward to."
"It’s almost as if they lay a path of breadcrumbs as they worked their way through the exam. " 
How do you think online proctoring has changed and/or will change the testing industry?
"All testing organizations want to meet the on-demand needs of their stakeholders and offer a positive exam experience. For some organizations, online proctoring satisfies both of these needs. Online proctoring has developed significantly since it was first introduced. Service providers have taken the time to address many of the security concerns online testing presents. Online proctoring can now include: comprehensive candidate authentication, challenge questions throughout the exam, keystroke analysis, and full recording of the test sessions, as well as live proctoring of the entire test session. These are serious approaches to test security challenges. Because of these services I think we’ll see more testing organizations consider online proctoring in the years to come.

"Online proctoring can be an especially good solution for programs with higher pass rates and a professional cohort of test takers. For example, right now we’re seeing many medical programs offer their maintenance of certification exams via online proctoring. This is a great option for their pool of test takers.  

"As their exams move to online proctoring, test security professionals will have to refocus their security plans, aiming to fill the gaps online proctoring leaves open. Those gaps will differ depending on the level of online proctoring services the program has in place. For example, programs that choose record and review sessions over live proctored sessions will need to develop procedures for what to look for during those reviews, what behavior flags will trigger a keystroke analysis, and how to respond to the test taker when something out of the ordinary is discovered during the review. Many of these procedures can be based on what they were doing for their brick and mortar administrations, but they will need to make adjustments. After a brick and mortar administration, for example, we can reach out to the test center administrator with questions about what she witnessed. With a record and review session, this won’t be possible. It will be important to think through how all available evidence will fit together and what the organization will do if they don’t have a complete picture of what happened during the exam."
One of the most exciting test security advances in recent years is the use of biometric authentication to confirm a test taker’s identity. Why do you think biometric authentication is a valuable security measure?
"At the CFA Institute, we use biometric authentication for our computer-based exam programs. It is a valuable way to confirm a test taker’s identify, especially for high-stakes programs which require the test taker to pass a series of exams. Biometric authentication allows us to confirm the same individual is taking the exam at each appointment.

"I do think biometric authentication will be challenged in the years to come as countries around the globe demand stricter privacy control for their citizens. It may be that the collection of a fingerprint or a palm vein scan proves too invasive under these new privacy standards. With the enforcement of the EU General Data Protection Regulation, we will see whether, or rather which, biometrics stand the test of time."
Technology can be used to steal and cheat, but it can also be used to help better protect tests. What role do you think technology will play in test security in the future?
"Technology presents both opportunities and challenges in every industry. In the testing world, dishonest test takers will try to benefit from technology advances, especially as tech gets smaller and more capable. The challenge for testing organizations is to identify the use of technology in the testing room. This is certainly easier to do in a brick and mortar testing room than it is with online testing.

"At the CFA Institute, our approach to combat nefarious use of technology in the testing room has been two-fold. First, we employ measures that identify tech devices on a test taker’s person before she enters the testing room. And of course, we use technology to make those identifications. As tech design advances, we expect the technology used to identify those advances will too. Second, we focus on training our proctors to look for unusual behavior during the administration of the exam. Even though technology is changing quickly, we find that test takers exhibit a series of specific behaviors when attempting to use technology in the testing room regardless of which device they are using. For us, it has been easier to identify that unusual behavior than it is to identify the tech device. We have been very successful with this approach."
"For us, it has been easier to identify unusual behavior than it is to identify the tech device that is being used."
Tests are increasingly administered across borders to include test takers from around the world. How might new technology and innovations help us securely give tests with this type global scope?
"At CFA Institute, all of our exams are global. We administer exams in more than 90 countries around the world, so this issue is near and dear to our hearts. We have been very happy to see how quickly test delivery vendors can provide item performance data and data forensics. The quick turn-around of this information allows a testing organization to identify issues before they get out of control. What goes hand-in-hand with that, of course, is the ability to rapidly publish new test forms or item pools to quickly combat threats that are identified. So, early detection of problems and a timely plan to identify and address those problems are vital; this is something technology has allowed."
How is the legal profession adapting to the ever-changing security threats?
"In a world with ever-changing security threats, it’s important for lawyers in the testing industry to come to the table with two things: (1) an understanding how intellectual property can be legally protected in the jurisdictions the program operates; and (2) an understanding of the privacy and cybersecurity laws in those same jurisdictions. This knowledge is essential to support testing staff in their efforts."
What is a time in your career when you have taken a leap into the unknown, embraced a new innovation, and experienced a positive change as a result?
"For me, data forensics is a great example of this. As a newbie in test security, I had no idea what I was doing. I’m not a psychometrician, so my understanding of data forensics and the algorithms used to identify unusual answer patterns did not come naturally. There was such a steep learning curve! Luckily, I started my testing career at ACT where I was surrounded by people who were passionate about psychometrics. They were also able and willing to take the time to explain processes; it is invaluable to work with a psychometrician who is capable of explaining what they do to a layperson. Now I’m a convert and a zealous advocate for all things data forensics." 
"When someone is comfortable with the status quo it is probably because they understand it and operate effectively within it. Embracing new ideas or a change to the status quo, means people have to learn something different."
Why do you think it is often difficult for people, both in our industry and in the wider world, to embrace change and accept new ideas?
"Almost everyone struggles with change, regardless of profession or industry. When someone is comfortable with the status quo, it is probably because they understand it and operate effectively within it. Embracing new ideas, or even a change to the status quo, means people have to learn something different. It means, at least in the short term, they may be less competent, effective and efficient at their job. That can be daunting."
In your opinion, what are the three biggest threats to test security right now?
"Every test security team has the same mission: to maintain the integrity of their exams by ensuring that exam content is secure; no candidate has an unfair advantage over another; only those individuals truly qualified to pass the exam do so; and the risks inherent in our programs are identified and addressed appropriately. Any threat to that mission is important.  

The three biggest threats to that mission revolve around pre-test exposure to exam content, candidate authentication, and theft of exam content. These overall threats remain constant for every testing program. What changes are the means dishonest players use to cause these threats. Those means are unique to the parties crafting them and the programs they are attempting to damage. This is why test security professionals need to be attentive to the weaknesses of their program and which individuals and organizations may be intending them harm."
You might also like...
More Reads
Can Good Items Have Bad Item Statistics? Use the latest scientific research conducted by Caveon to evaluate all of your options and make the best decisions for your testing program.
Can Good Items Have Bad Statistics?
Data-Driven Success:
toc_icon_BLUEcaveon
caveontoc_icon_BLUE_desktop
Contact
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.
Submit
Join our mailing list
Copyright© 2018 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
caveontoc_icon_BLUE_desktop
caveon
Home
Featured
Short Reads
Contact
  • The Future of Multiple Choice
  • ATP Women Give Back
  • Ask an Expert: Aimée Rhodes
  • Name It To Tame It
  • Industry Insider: Jim Lucari
  • SmartItems and AIG Compared
  • Can Good Items Have Bad Item Statistics?
  • Celebrating 15 Years: An interview with Jamie Mulkey
  • Ask Agatha
download-button
Download PDF
cross_close_BLUE
  • Caveon Announcements
caveontoc_icon
Home
home_icon
Featured Articles
star
→ The Future of Multiple Choice
→ ATP Women Give Back
→ Ask an Expert: Aimée Rhodes
→ Name It To Tame It
Short Reads
bookicon2
→ Industry Insider: Jim Lucari
→ SmartItems and AIG Compared
→ Can Good Items Have Bad Item Statistics?
→ Celebrating 15 Years with Jamie Mulkey
→ Ask Agatha
→ Caveon Announcements
Contact
envelope_white
cross_closecaveon