Smartphone assessment equivalence: A South African study

In today’s article, we examine the important question of smartphone assessment equivalence as applied to the local South African market.

Smartphone assessments are an increasingly important tool in IO Professionals’ toolkit, especially for those who serve organisations that are interested in expanding their talent pools, engaging Millennial job-seekers, and accessing some of the unique benefits that smartphone technologies have to offer.

As we’ve discussed in previous articles, smartphone assessments are objective assessments that use the capabilities of mobile technologies and modern coding standards to play assessments on smartphone devices.

Smartphone assessments are designed to be adaptive, so candidates can elect to complete their assessments on tablets or traditional laptops / desktops. The publishers of such assessments focus on ensuring a common experience across devices, with the related goal of ensuring equivalence of measurement irrespective of the device used to complete the assessment.

Much international research has shown that there are no significant differences between scores obtained from smartphone assessments and assessments completed on other devices (e.g. laptops). This is especially true if these assessments are designed with resolution and screen size optimisation for smartphones in mind. However, local research illustrating similar findings in South African samples has been somewhat lacking.

Our study

Our research team at TTS recently completed a study investigating equivalence among selected cut-e (an Aon company) competency assessments.

We used a graduate sample (N=166) within the industrial and mining sector, and investigated both result equivalence as well as participant perceptions and experience of the assessments used.

The assessment used (and their domains of measurement) were:

  • motionChallenge: Complex planning capability
  • scales ix: Discovering rules to specific problems
  • scales lst: Applying known rules to specific problems
  • scales numerical mobile: Analysing complex numerical data
  • scales verbal mobile: Analysing complex verbal information
  • switchChallenge: Exploring and applying rules to specific problems

In the study, 67 participants completed their assessments using laptops or desktops, 54 participants completed their assessments using smartphones, and 45 participants used a combination of laptop and smartphone for specific assessments.

Results: Equivalence of measurement

The graph below shows some of our findings regarding equivalence of assessment results:

Note: All above differences non-significant (Cohen’s d between -0.2 and +0.2; i.e. Small to negligible effect sizes

As shown in the graph above, none of the assessments in the study showed significantly different results when compared across devices.

This suggests that at least for the assessments shown, there are few reasons for practitioners to be concerned about equivalence between laptop and smartphone-delivered measures.

Results: Equivalence of participant experiences

We were also interested in how participants experienced the assessments and wanted to investigate if the device used (i.e. smartphone vs. laptop) affected their perceptions of fairness, ease-of-use, applicability and other user factors.

Below is a graph showing the results across a number of perceptual and experiential categories:

As in the results findings, there were no significant differences in how participants experienced smartphone versus laptop assessments. This indicates that for most of our participants, the device used to complete their assessments was largely irrelevant to their test-taker experience.

Concluding thoughts

This study of smartphone assessment equivalence discussed above is part of an ongoing project at TTS to investigate new assessment technologies. We are especially interested in bringing best of breed products to the South African market and our local clients.

Smartphone assessments offer several benefits to organisations:

  • They allow clients to access a larger potential talent pool, especially in countries like South Africa where more people have access to smartphones than laptops or tablets.
  • In addition, they allow for maximum test-taker choice, since candidates can elect to complete their assessments on the device they prefer.
  • Also, having smartphone assessments available helps organisations with technology-forward brand promises to align their recruitment and selection practices with the latest in mobile technologies.

If you’re interested in using smartphone assessments in your next project, why not give us a call or contact us at: info@tts-talent.com.