In recent times, talent professionals have had to adapt their approaches to the new reality of remote working, social distancing, and a tightened economy. In such a business landscape, the use of talent assessments, especially assessments that are delivered online and unsupervised, has become especially important.
In this article, we address some of the potential concerns that may be raised in the use of such assessments and examine best practices in IOP when delivering online assessments.
Common misconceptions about online assessments
Like any new technology introduced within the IOP profession, online talent assessments were initially treated with skepticism. This is appropriate: Given the ethical responsibilities of IOPs and talent professionals in the workplace, innovation must be tempered by best practices and care for the organization’s and individual employee’s wellbeing and success.
Common concerns raised in regard to online, unsupervised assessments have taken predominantly one of three forms:
- Concerns regarding honesty and cheating during completion of unsupervised assessments. In other words, can the results from unsupervised assessments be trusted?
- Concerns about data integrity and comparability of results between traditional supervised assessments and unsupervised, online assessments. In other words, are unsupervised assessments equivalent to their supervised counterparts?
- Concerns around fairness of online, unsupervised assessments. In other words, do such assessments run the risk of disparate treatment of candidates?
Let’s look at each of these in turn.
Concerns about honesty and cheating
A common question we routinely field in our application of online, unsupervised talent assessments to client challenges is whether candidates won’t be able to easily cheat or be dishonest in their responses.
This seems logical: Surely the lack of supervision allows for cheating more so than if such assessments were supervised?
The reality however, is quite different.
Research on cheating behavior as well as comparability of supervised assessments with unsupervised counterparts reveal that there are no significant differences found in the results of such assessments (see for instance, Bartram and Brown, 2004). Results like these imply that incidences of dishonest responses on assessments are not dependent (or controlled for) by supervision per se.
Although the topic of dishonest behavior is a difficult one to research directly, available evidence suggests that cheating behavior is far less common than might be assumed, and that candidates who intend to cheat during assessments will not necessarily be perturbed by supervision.
Also, given the timed nature of many aptitude assessments, any outside help that a candidate might receive is more likely to harm than enhance performance.
Strategies for the mitigation of cheating risk
Although the incidences of cheating during online, unsupervised assessments are likely less than is feared, IOPs and talent professionals can apply several strategies to mitigate the risk:
- Using parallel forms: assuming that the measure has a large item bank, participants can be assessed using parallel forms that will either mitigate against memorizing of compromised item answers as well being used to verify a participant’s initial scores.
- Modern statistical methods like Item Response Theory can be used to calibrate items and provide early warning of potential item compromise (e.g. an item shows a sudden deviation from expected responses or known difficulty).
- Controlled access: given that online assessments are delivered using platforms optimized for testing, only filtered candidates will have access to test batteries and have a limited time available in which to complete the assessments.
- Honesty contracts and warnings: before accessing the assessments, candidates can be given extensive warnings regarding verification or follow-up assessment, as well as information about the consequences of cheating or breeching security. In addition, candidates can be requested to complete honesty contracts, which require them to digitally agree to behave ethically. Research suggests that such measures can do much to curb dishonest behavior.
A far greater point of concern is the one of test item compromise. When test items and answers leak into the public domain, there is a real threat to the integrity of the affected assessment’s results. Unscrupulous candidates might well purchase or obtain such items in preparation for their assessments.
But this is exactly where online assessments, especially those produced by best-of-breed, internationally recognized providers come into their own. At TTS, one of our persistent criteria for adopting an assessment product is that it should feature technological measures that prevent the risks associated with item compromise.
Usually, this will take the form of some form of item banking.
Item banking is a unique feature of certain online assessments where large libraries of items, shown to be comparable in terms of measured construct, and in the case of aptitude tests, difficulty, are stored in the product provider’s database. Using this technology ensures that assessments can be dynamically composed, thus preventing candidates from receiving the same assessments. In addition, items are constantly analyzed for deviance from the expected norm and removed if found to be potentially compromised.
This is only possible if the provider has (a) an extensive library of comparable items, (b) has done the needed item response analyses to classify items based on difficulty and (c) has invested in technology that allows for a seamless user experience as well as removal of potentially compromised items.
Such features are generally absent from offline assessments as well as online assessments that have only one set of questions. As experts in the field of talent assessments, we see it as our role to advise our clients about best-of-breed product providers that are able to overcome these limitations.
Concerns about comparability of results
As already mentioned in the previous section, research conducted on comparability of supervised versus unsupervised online assessments have overwhelmingly found few significant differences between these two methods (see for instance, Bott, et. al, 2004; Inceoglu & Bartram, 2006; Joubert and Kriek, 2009; Sinar & Reynolds, 2004).
In this regard, and because of the overwhelming evidence available, international scientific associations like SIOP largely consider the debate on this matter to have been resolved in favour of the notion that there are no significant differences in validity between proctored versus unproctored assessments.
To summarize these findings, we can conclude that:
- There is wide-ranging test equivalence between modes of administration (i.e. proctored versus unproctored, online versus offline) for cognitive and personality-based assessments.
- These measures have comparable psychometric properties (e.g. construct validities, reliability, etc.).
- The same norms can be used for both conditions.
And moving into the more mobile-enabled world, TTS researchers have recently found comparability of results even between PC-delivered assessments and their smartphone equivalents.
Given these findings, it seems that any differences in equivalence between supervised and unsupervised assessments are non-significant and pose no barrier to their continued use for talent assessments.
Concerns about testing fairness
Of course, the preceding section answers at least partially the question of fairness. If psychometric properties of assessments are not affected by their administration mode, they are likely to be fair across participant pools.
But an additional concern has been one of access to online technologies. In this sense, there may be an argument for inviting candidates to a supervised venue in order for them to have access to the needed technology to access the testing opportunity.
But this concern is largely unfounded if one considers the very wide penetration of mobile and smartphone technologies in countries like South Africa. Increasingly, most potential candidates that could conceivably be targeted in a recruitment drive will have access or know someone with access to a smart device.
And given the comparability of results between devices, access is actually likely to be more wide-ranging and greater for such assessments when compared with the traditional, supervised approach.
That is because smartphone-delivered, unsupervised assessments do not depend on specific geography, but rather access to the appropriate technology and internet coverage. As recent Google consumer reports reveal (Google Consumer Report, 2018), countries like South Africa enjoy very widespread access to mobile, internet-based technologies, especially among the Millennial generation.
Indeed, the use of online, unsupervised assessments is also perceived by such candidates as a fair process, and potentially lends esteem to the recruitment and assessment project (see for instance, Dowdeswell, 2006).
Final thoughts
The IOP profession holds a privileged position in most organizations it serves. In many ways, IOPs are the custodians of scientific thinking within organizations as well as the use of objective measurement to help managers to make more ethical and accurate decisions about talent.
Consequently, we must take care to thoroughly evaluate new technologies that are introduced into talent selection decision-making. Given the above discussion and increased evidence that suggests no differences in equivalence between online, unsupervised assessments and supervised counterparts, we feel confident that mandating the use of unsupervised, online assessments is not only scientifically warranted, but represents current IOP best practice.
Using such assessment products and methods allow for a far wider use of assessments through online (and mobile) delivery, thus giving more access to objective assessments for even candidates who are very remote. In addition, ensuring the use of technologies such as item banking effectively deals with real threats to assessment integrity.
Using unsupervised, online assessments represents an opportunity for IOPs to deliver objective assessments in a faster, more efficient, and more accurate way than ever before.
For instance, time-to-hire can be decreased significantly, making the process more efficient for both hiring managers and candidates.
In the new normal of greater use of online meeting platforms, remote work, and digitization of business practices, IOPs need not limit themselves artificially to only using supervised assessments. Instead, the use of robust, best-of-breed online products will significantly benefit the businesses they serve.
If you would like to know more about TTS’s portfolio of online talent assessments, email us at: info@tts-talent.com
Sources & suggested reading
Bartram, D., & Brown, A. (2004). Online testing: Mode of administration and the stability of OPQ 32i scores. International Journal of Selection and Assessment, 12(3), 278-284.
Joubert, T., & Kriek, H.J. (2009). Psychometric comparison of paper- and-pencil and online personality assessments in a selection setting. SA Journal of Industrial Psychology, 35(1), 1-11.