SailPoint is a leader in
identity governance
which plays a critical role in modern enterprise security. One of
SailPoint’s core values
is “Innovation” or developing creative solutions to real challenges. This past year, SailPoint teamed up with Caveon to do just that. In this brief article, we describe the joint, successful implementation of the new testing technology of SmartItems
for the first two SailPoint certification exams, both released in 2018. SailPoint certification program staff had been aware that IT certification programs can have their exams stolen within days of being released. Those exams end up on braindump sites, making test scores from those exams useless for certification decisions. SailPoint wanted to ensure that any certification exams bearing its imprimatur would be as secure as the available testing technology would permit. Therefore in 2018, in its first certification program ever, SailPoint decided to use Caveon SmartItems for 100% of each exam. SmartItems are unique in that each one renders in thousands or even millions of ways, providing a different but fair experience for each test taker. SmartItems are invulnerable to theft and cheating, relieving SailPoint of the same security concerns that have plagued every other IT certification program for the past three decades. SmartItems brought at least two other bonus advantages to SailPoint. The first is that SmartItems motivate proper preparation. Brain dump material is unavailable, and multiple-choice-test-taking skills are irrelevant; to pass the exam they need to study and gain experience on SailPoint technologies. The second advantage is in cost savings. Because SmartItems can’t be compromised, they will only need to be replaced because of legitimate performance issues or inevitable content updates, not because a question or even an entire form was stolen.
You might also like...
More Reads
Worth more than a pound of cure, here are six preventative measures that any testing program can implement to reduce security headaches.
Read more →
Test security expert Jennifer Geraets shares her wealth of knowledge on effective exam design features, valuable security measures, and exciting innovations to help strengthen your program.
Read more →
Join our mailing list
Copyright© 2019 Caveon, LLC.
All rights reserved.
Privacy Policy
Terms of Use
Interested in learning more about how to secure your testing program? Want to contribute to this magazine? Contact us.
The Prevention Lockbox
Arthur Altman, SailPoint
David Foster, Ph.D., Caveon
The Use of
SmartItems on
Certification Exams
About the Authors

Arthur Altman

Arthur Altman is a Program Manager in the Global Education Services business at SailPoint Technologies Holdings, where, among other duties, he stood up and now oversees the SailPoint Certified Identity Professional Program, including the launch of the first-ever certification exams at SailPoint in 2018, with more coming in 2019.
David Foster, Ph.D.

A psychologist and psychometrician, David has spent 36 years in the measurement industry. Over the course of his career, Dave’s background in research and innovation has motivated him to apply a scientific approach to addressing problems in testing while continually searching for new and innovative solutions for improvement.
“SmartItems are invulnerable to theft and cheating, relieving SailPoint of the same security concerns that have plagued every other IT certification program for the past three decades.”

1. We learned that SmartItems can be congruent with and built to very complex SailPoint objectives.

2. We learned that the most important SME task isn’t to write items, but to design SmartItems to completely cover the objectives.

3. We learned that SailPoint SME’s, working in small teams, were able to utilize Caveon coders and technology to create SmartItems efficiently.

We learned that SailPoint SMEs, who were all novice item writers, enjoyed writing SmartItems. Because SmartItems afforded more creative options for covering objectives, for instance, enabling as many correct options and incorrect options as needed to cover a given item, item authors could follow the subject matter wherever it took them. This was fun!

5. We learned that the SmartItem authoring tools would need correspondingly functional item review capability, to enable authors to more quickly yet fully check item content and design.

6. We learned that, despite the newness of the SmartItem approach, it took about the same amount of overall time and effort to create SmartItems as it would have taken to create traditional items.
Of course, the biggest benefit of SmartItem use is that test scores can be trusted, forever, for certification decisions. With each test administered, SailPoint can be confident that candidates who pass the test are competent in the skills required by SailPoint. As long as the certification program is in place, the exams’ test scores can be used to make reliable certification decisions. The process of creating and publishing certification exams is an extensive one, and one that needed some modification for SmartItems. Since two exams have been published and made available to candidates, what have we learned?
Field Testing

A field test of the SmartItems (also sometimes called a “beta test”) was conducted to see if SmartItems performed well psychometrically ahead of exam release.

1. We learned that, using established psychometric standards, SmartItems perform as well as or better than traditional items.

2. We learned that SmartItems contribute to the high reliability of SailPoint exams.

We learned that SmartItems contribute to the gathering of evidence for the valid use of test scores for the SailPoint exams.

4. We learned that we could create pass/fail cut scores easily for tests using SmartItems.

5. We learned, in general, that there is no psychometric reason to not use SmartItems in exams.

In conclusion, we learned that the promised benefits of SmartItems, including stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation, are confirmed.

Because of the very positive experiences from 2018, a third exam using SmartItems is underway and is expected to be released in the first half of 2019.