March Edition
Detecting Anomalies
Learn about using data forensics analyses to detect instances of test fraud in K-12 test administrations.
Data Forensics analyses are used to identify statistical anomalies and testing irregularities that are indicative of potential test fraud. It is recommended that these analyses be performed after every test administration in order to identify schools, teachers, test administrators, and students who might be associated with invalid scores and results. Data forensic analyses should be supported by test delivery, scoring, and other systems that will capture and store appropriate test response data elements to make it possible to employ the detection statistics.
Data forensics analysis includes the generation of various statistics designed to identify whether various test security breaches may have occurred. These analyses can be targeted to the specific threats identified by the state. Test security threats in K-12 assessment programs that can be detected by data forensics statistics include:
  • Test-takers who may have communicated with each other during the exam
  • Student test-takers who share answers
  • Teachers or other proctors who disclose the actual test questions, or allow proxy-test taking
  • Test content that may have been exposed prior to giving the test
  • Test-takers who may have been coached or received unauthorized assistance
  • Coaching of actual test content
  • Disclosure of actual test content by an individual or on the Internet
  • Inappropriate tampering of test materials, or inappropriate direction during testing

States should identify which test security threats are of concern; additional threats to those listed should also be considered. Vendors should indicate and list the analyses that will be used to detect the threats that the state has identified, including unique aspects of the state assessment program, the principles that govern forms construction and how the test will be administered. For example, when computers are used to administer tests, data in addition to selected responses may be collected and used. Examples of these additional data are to include when a student suspended the test to take a break, how much time was spent answering questions, whether students have changed answers to the questions and how often, the time of day when the test was administered, and/or the time when each question was answered. When available, these additional data elements can be processed and analyzed to detect potential test security violations. States should detail these data elements and require responses from vendors as to whether these data will be collected and how they may be analyzed. Data forensics analyses should provide information about individual students, educators (through use of classroom and aggregated data), schools, and the exams. States should understand how data forensics analyses will be conducted to provide this information.
Use of Data Forensics Results
The data forensic analysis results should be sent to the state for review, should include recommendations for the next steps, and should inform what appropriate responses may be invoked. A tight turn-around is often necessary to meet scoring and reporting deadlines following each test administration. A procedure should be established for flagging identified scores with an appropriate status such as “indeterminate,” “hold,” or “invalidation,” based upon the SEA’s policies and procedures.
Data forensics analyses can serve as quality assurance to ensure that policies and procedures are being implemented. Student flags may indicate the need for additional training for test administrators; school flags may suggest candidate sites for on-site monitoring in future administrations.
Quality Assurance
Data forensics analyses can serve as quality assurance to ensure that policies and procedures are being implemented. Student flags may indicate the need for additional training for test administrators; school flags may suggest candidate sites for on-site monitoring in future administrations.
What to Expect
States should spell out the requirements for data forensics to ensure that vendors provide the level of detail and statistical analyses, reports, and report interpretation needed to support the state's policies. These requirements and interpretations should include whether score invalidations will be used, and what reporting timelines are needed to meet the state’s desires. Data requirements for handoffs between prime vendors and potentials subcontractors should be identified and defined by the vendor(s). States should also understand the specific types of analyses required in relation to the anticipated security threats. Attention should also be paid to the modes of the administrations of interest (e.g., paper/pencil, computer-based, etc.) Specific analyses should be carried out that would indicate non-independent test-taking between pairs of students, collusion among groups of students or proctors and students, and prior exposure to test content. Data forensics analyses should also include an investigation of unusual score gains or losses across classrooms and schools. Vendors should also describe the process for working with the state to communicate data forensics results.
You might also like...
More Reads
Monitoring of test administrations can provide valuable information on the fidelity with which the test administration and security procedures are being applied.
Onsite Test Administration Monitoring
Featured Articles
→ The Language of Test Security
→ It's a Risky Business
→ Jim Wollack Speaks
→ 5 Steps to K-12 Test Security Success
Short Reads
→ Interview: John Fremer
→ Industry insider: 30 Questions for Walt Drane
Interested in learning more about how to secure your testing program? Contact us.
Join our mailing list
Copyright© 2018 Caveon, LLC.
All rights reserved. Privacy Policy | Terms of Use
Featured Articles
→ The Language of Test Security
→ What Are My Threats
→ A Tale of Two Cities
→ Web and Social Media Monitoring
Feature Articles
→ Detecting Anomalies
→ Onsite Test Administration Monitoring
→ Investigating Testing Irregularities
→ Resources
→ About Caveon