The debate regarding supervised (i.e. proctored) and unsupervised talent assessments is familiar to many in the industry. In previous articles, we have examined evidence supporting the use of online, non-proctored assessments.
The results from these landmark studies are clear: supervising candidates while they complete their assessments, whether such assessments are ability tests or behavioral measures, does little in the way of affecting assessment results when compared to unsupervised completions.
As a result of such findings, online, non-proctored assessments have become the norm in the IO Psychology and talent management industries. However, new proctoring technologies using Artificial Intelligence (AI) have recently made headlines, especially in higher education applications.
What is AI Proctoring?
AI proctoring may be used either concurrently (i.e. live, in the testing session itself) or asynchronously (i.e. based on a recording of the testing session). In either case, AI algorithms are used to analyze candidates as they complete online assessments, using their computer’s webcam and internal sensors. Data analyzed include the candidate’s face, eye movements, computer screen as well as input actions (e.g. mouse movements and keystrokes).
Such algorithms are trained to detect human faces (facial recognition) and ideally, can match candidates to pre-registered identification photos. In addition, the AI can compare the candidate’s computer screen and actions to a set of known suspicious behaviors such as using other browser windows or receiving help from someone off-screen.
Depending on the software, AI proctoring will then either:
- Send a post-testing report to the client or testing company about whether a candidate is likely to have completed the assessment appropriately or,
- Intervene directly to block the candidate from completing the rest of the test until appropriate actions have been taken.
Challenges with AI Proctoring
To date, the widest use of AI Proctoring has been in educational settings such as high schools and university exams that take place remotely. In response to the COVID-19 pandemic, several companies began to offer such services to leading educational institutions to aid in the remote educational model many of them had adopted.
Unfortunately, the use of AI Proctoring has not been without controversy. The now well-known challenges AI has had in recognizing faces of people of color was one significant problem. Minority students were often blocked from starting assessments because the AI algorithm could not recognize their faces sufficiently to compare to reference photos.
In addition, some AI Proctoring services seem overly vulnerable to false negatives: Excluding students from completing their exams based on innocuous behaviors or features of their immediate environments (e.g. having textbooks in the background of their webcam video feed).
AI Proctoring and talent assessments
Despite the above challenges, AI Proctoring is an attractive technology to assessment providers for a few reasons:
- Unlike individual remote proctoring using existing meeting software applications, AI Proctoring can be used to screen multiple candidates concurrently.
- AI Proctoring could help detect or prevent the capturing of test items by candidates who are competing assessments under false auspices.
- The technology can augment or replace the already-established practice of re-testing candidates with equivalent form assessments when cheating is suspected.
Because of the challenges mentioned above, test publishers have been understandably cautious in their approach to AI Proctoring.
Saville Assessment, for instance, recently published an article calling for the necessity of using AI Proctoring in conjunction with human auditing (see Chan, 2024). In cases where AI Proctoring is used, it is advisable that humans review the results of such proctoring, and that human experts make the final decision regarding a candidate’s likelihood of having engaged in dishonest test behaviors.
This obviously assumes that the AI Proctoring technology being used is a post-testing instance and not one that is applied live, in the testing session.
And even in cases where dishonesty is suspected, Saville recommends that the candidate be re-tested using a more individual, human-mediated proctoring technology rather than labeling them as “dishonest”.
AI and non-AI Proctoring: Do we need it?
In higher education, AI Proctoring has had an inconsistent track record. While it is probably here to stay, many schools and universities in the USA and other countries have elected not to use AI Proctoring until more reliable and unbiased technologies are developed.
Within talent assessments, the picture is somewhat different.
For one, there is already a well-established practice of unsupervised, online assessment for selection or development. The data supports this practice (see for instance, Joubert & Kriek, 2009), but more so, global test publishers have gone to great lengths to make their assessments near cheat-proof (see our article on this topic).
Such measures include:
- Using large test item banks to ensure that revealed items will have a negligible effect on test integrity.
- Constant analysis of test items for inconsistent performance (i.e. items performing as “easier” than expected) and replacement with new items in the item bank.
- Dynamic, on-the-fly test construction using such item banks to ensure that no candidate ever completes the same test twice.
- Sophisticated penetration testing and data security practices that ensure test publisher databases and test items are protected from hacking.
In the case of aptitude assessments, their timed nature ensures that any help received will almost certainly adversely affect a candidate’s score.
All of the above raises the question of whether AI Proctoring is needed in talent assessments. Given the imperfect nature of AI Proctoring technologies, combined with the risk of false negatives and consequent negative impact on a candidate’s test-taking experience as well as potential reputational impact for companies using such practices, caution seems to be a prudent approach at present.
Final thoughts
In previous articles, we have examined measures implemented by global assessment providers in countering common sources of cheating and dishonesty. In addition, emergent technologies such as Large Language Models (LLMs) have been identified as potential vectors for dishonest test-taker behavior (although at present, their threat to the integrity of psychometric assessments seems low).
AI Proctoring represents the other side of that coin: using AI to detect or prevent dishonest test-taker behaviors.
While some elements of AI Proctoring may have labor-saving benefits, the technology as currently conceived is not without risk. This problem seems especially noteworthy given the already well-developed methods test publishers have at their disposal for preventing cheating, along with research data that shows little difference between supervised versus unsupervised testing, irrespective of the mode of supervision applied (i.e. online or in-person).
This leads us to ask the question in the title of this article: is AI Proctoring perhaps much ado about nothing? We recommend caution and look forward to seeing how this technology impacts the future of our profession.
If you would like to know more about this topic or other assessment best practices, contact us at info@tts-talent.com
Sources and further reading
Brown, A., Bartram, D., Holtzhausen, G., Mylonas, G. & Carstairs, J. (2005). Online personality and motivation testing: Is unsupervised administration an issue?Paper presented at the 20th annual SIOP conference, LA.
Caplan-Bricker, N. (2021). Is online test-monitoring here to stay?. The New Yorker, NY.
Joubert, T., & Kriek, H.J. (2009). Psychometric comparison of paper and-pencil and online personality assessments in a selection setting. SA Journal of Industrial Psychology, 35(1)
Nigam A, Pasricha R, Singh T, Churi P. (2021). A Systematic Review on AI-based Proctoring Systems: Past, Present and Future. Education and Information Technologies, 26(5), 6421-6445.
Chan, S. (2024). Discussing remote test supervision at the 2024 ITC Conference. Retrieved at: https://www.savilleassessment.com/remote-test-supervision/
Tippins, N.T., Beaty, J., Drasgow, F., Gibson, W.M., Pearlman, K., Segall, D.O. & Shepherd, W. (2006). Unproctored Internet testing. Personnel Psychology, 59, 189-225.