ADEPT-15® is a scientifically-based assessment designed to accurately uncover the unique aspects of an individual’s personality to hire, promote, and develop the very best talent that fits with an organization’s culture.
The profile from the questionnaire can be used to inform decision-making during selection, promotion, or development across all different levels of expertise and management.
ADEPT-15® has been proven by over 8 million global administrations, is backed by 50 years of research, and is built upon a database of 350,000 unique items.
In this article, we discuss some of our research team here at TTS-Top Talent Solutions’ recent work on validating Aon’s Assessment Service’s award-winning measure of workplace personality, the ADEPT-15.
Key takeaways that will be discussed in more detail below are:
- The ADEPT-15 is registered with the Health Professions Council of South Africa
- Norming research has established that there are negligible differences between performance of South African candidates on the international norm, making it appropriate for local use
- Based on our research within low and high stakes assessments, the ADEPT-15 shows robust psychometric properties in terms of reliability and validity
- Our data also supports the ADEPT-15 as a fair measure that contains no apparent bias based on gender or race.
For a more detailed look at the personality model and international pedigree of the ADEPT-15, you can read our overview article by clicking here.
Registration as a psychological instrument
As a measure of personality, ADEPT-15 was classified by the Health Professions Council of South Africa as a psychological test in April 2021. A copy of the confirmation letter is included in the South African Research Supplement and is available on request from our product team here at TTS.
Regardless of whether an assessment is psychological in nature or not, as a condition of access TTS requires that an HPCSA-registered psychologist or psychometrist controls the use of occupational assessments in an organization.
This includes decisions regarding test choice, how to administer the tests, selecting norms, setting cut-off scores, and who may interpret and provide feedback.
To this end, we’re currently offering international accreditation training in the use of the ADEPT-15 in the workplace for registered professionals. To book your seat, click on this link.
Social desirability data
Given the fact that ADEPT is an adaptive questionnaire that will present test-takers with different item sets based on how they respond to items, it is important to evaluate the social desirability of items in the South African Context. Items are presented in pairs and if the social desirability of items is different in the South African context, the responses would also be distorted, and candidates might endorse highly socially desirable items more than undesirable items. This will then cause distortions.
In order the evaluate the criticality of the items’ social desirability levels in ADEPT-15’s our research team at TTS partnered with Aon to conduct a social desirability evaluation of all the items in the South African context.
The goal of this project was to establish the social desirability of each ADEPT-15 item within the South African context and therefore ensure that items could accurately be paired, adaptively based on candidate responses and IRT, in the assessment.
To do this, 12 participants – registered psychologists and intern psychologists familiar with the principles of personality assessment, South African citizens or long-time residents, from different cultural backgrounds and languages, that are proficient in English – were recruited for the investigation.
Each participant evaluated all 1470 ADEPT-15 item statements based on each statement’s level of social desirability in South African working culture.
A high level of agreement and consistency was found between participants’ ratings, and the results illustrated that the social desirability values determined for South Africa were broadly similar to the United States version, where the ADEPT-15 was originally developed.
In cases where differences in the desirability of items were observed, they were in line with hypothesized cultural differences between US and SA samples. The local social desirability data will therefore be used to minimize response distortion in local candidate pools.
Norming research
To provide guidance concerning norm choice in the South African context, we compared local data to Aon’s International General Adult Population Norm (N=40,390). The local dataset of 630 participants was compiled from a series of trials conducted with research volunteers, training delegates, and applicants, carried out in South Africa between June 2019 and November 2020.
Only two of the 15 dimensions in the ADEPT-15 model showed potentially meaningful differences, namely Cooperativeness (where the international norm is just over half a standard deviation higher than the South African sample), and Humility (where the South African sample is just over half a standard deviation higher than the international norm).
The remaining dimensions as well as the consistency measure show small to negligible differences, with the absolute value effect sizes ranging from 0.02 to 0.48, with a median of 0.16.
Given the similarities between the South African dataset and the international norm, we would recommend that local practitioners use the international General Adult Population norm. None of the dimensions showed differences of more than half a standard deviation on the Stanine scale and thus results will be a robust benchmark for South African candidates.
We are however continuing to add our local data to the international global data set as the ADEPT gains wider local use which will make the international norm even more representative.
This finding is in line with most other international research around the development and use of personality scales globally. Most studies found very small sub-group differences between countries on occupational or work-based personality questionnaires. In general, these types of questionnaires tend to travel very well.
Reliability and validity data
The test-retest approach is the most appropriate form of reliability for assessments like ADEPT-15 and indicates the extent to which the constructs measured by ADEPT-15 are measured consistently over time.
Our study calculated test-retest reliability estimates for a sample of South African participants who first completed the ADEPT-15 assessment as training delegates or research volunteers. After a period of two to three weeks, participants were invited to complete the assessment again, specifically for the purposes of calculating test-retest reliability, yielding a final sample size of 200.
In the initial analysis, while the range of reliability estimates obtained was similar to the study reported in the Aon technical manual (Martin & Theys, 2019), the median estimate for the South African sample was slightly lower. However, the fact that the sample comprised both training delegates and research volunteers who completed their assessments in a low-stakes environment may have had an influence on the lower estimates observed. When utilizing the Consistency score to remove participants who may have been less attentive in their responses, the results of this study then show benchmark aligned reliability for the measure, suggesting stability of results across time.
| Test-Retest Reliability Estimates | United States Sample (N=112) | South African Sample (N=200) | Screened SA Sample (Consistency > 6; N=113) |
| Median Correlation | 0.66 | 0.57 | 0.64 |
| Range of Correlations | 0.44 – 0.73 | 0.42 – 0.70 | 0.41 – 0.77 |
Another consideration is that traditional Likert-type personality questionnaires are likely to overestimate the reliability of constructs due to the influence of socially desirable responses (Martin & Theys, 2019). Given that the design of ADEPT-15 removes much of the impact of socially desirable responding, the observed test-retest reliabilities may still be indicative of good reliability despite being lower than expected of more traditional Likert-based measures.
To date, our examination of the validity of ADEPT-15 in the South African context has been focused on construct validation, with recruitment for local criterion validation partners underway.
In the interim, several international studies supporting the predictive validity of the instrument are included in the Aon technical manual (Martin & Theys, 2019).
Four construct validation studies have been conducted in South Africa, two within low-stakes (training) contexts and two within high-stakes (selection) contexts, where ADEPT-15 was administered alongside a selection battery but not used in decision making.
| Aon’s ADEPT-15 & Shapes (Management) Questionnaires | Low-stakes Study (2020) | High-stakes Study (2021) |
| Median of significant a priori correlations | 0.24 | 0.20 |
| Range of significant a priori correlations | 0.13 – 0.55 | 0.13 – 0.46 |
| Aon’s ADEPT-15 & Views Questionnaires | Low-stakes Study (2020) | High-stakes Study (2021) |
| Median of significant a priori correlations | 0.14 | 0.18 |
| Range of significant a priori correlations | 0.13 – 0.15 | 0.15 – 0.19 |
In general, the results of the South African studies were aligned with expectations and followed a similar pattern of results as compared to international construct validation studies reported in Aon’s technical manual:
- In the convergent construct validation studies concerning the ADEPT-15 and Shapes (Management) questionnaires, most of the expected relationships were observed and where unexpected relationships were observed, reasonable explanations were identified.
- In the discriminant construct validation studies concerning the ADEPT-15 and Views questionnaires, relatively few significant relationships were observed, and those correlations that were statistically significant were only of small effect size.
Taken together, the results are largely supportive of the ADEPT-15’s convergent and discriminant validity and provide further evidence for its construct validity, particularly for South African candidates.
Considerations of test fairness
The potential for group differences in ADEPT-15 dimension scores was analyzed by gender and ethnicity, using the same sample as the initial South African trial norm investigation (N=630). Participants reporting English as their first language were also compared to participants whose first language was not English.
| Absolute Value Effect Sizes | Gender (Males = 143; Females = 487) | Ethnicity (Black = 284; White = 338) | English First Language (EFL = 179; ESL = 444) |
| Median d-statistic | 0.10 | 0.10 | 0.09 |
| Min d-statistic | 0.00 | 0.01 | 0.02 |
| Max d-statistic | 0.41 | 0.27 | 0.20 |
Overall, the analyses indicated minimal differences between the groups, consistent with established research particularly amongst race/ethnic, age, and gender groups using personality instruments (Hough et al., 2001; Ones & Anderson, 2002), providing support for the use of the ADEPT-15 assessment in the South African context.
The minimal differences observed between groups of interest also suggest that the inclusion of the ADEPT-15 in assessment batteries alongside cognitive ability tests will likely reduce adverse impact in line with local and international research on this topic (Kriek & Dowdeswell, 2010; Sackett et al., 2001).
Final thoughts
We are always excited to bring international best-of-breed assessment products to our local clients and with the ADEPT-15, we have an award-winning personality measure that is sure to benefit talent decision-making while furthering the technological and psychometric advancement of South African IOP practice.
We will accumulate more data on the ADEPT-15 questionnaire in the South African context and will make this available to users and clients on an ongoing basis.
If you would like to more about the ADEPT-15, why not attend our accreditation training or drop us a line at info@tts-talent.com.
References
Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned. International Journal of Selection and Assessment, 9(1–2), 152–194. https://doi.org/10.1111/1468-2389.00171
Kriek, H. J., & Dowdeswell, K. E. (2010). Adverse impact in South Africa. In J. L. Outtz (Ed.), Adverse Impact: Implications for Organizational Staffing and High Stakes Selection. Routledge/Taylor & Francis Group. https://doi.org/10.4324/9780203848418
Martin, N. R., & Theys, E. (2019). ADEPT-15: Technical Documentation (2019 Edition; 3rd Editio). Aon Human Capital Solutions. www.aon.com
Ones, D. S., & Anderson, N. (2002). Gender and ethnic group differences on personality scales in selection: Some British data. Journal of Occupational and Organizational Psychology, 75, 255–276. www.bps.org.uk
Sackett, P. R., Schmitt, N., Ellingson, J. E., & Kabin, M. B. (2001). High-stakes testing in employment, credentialing, and higher education prospects in a post-affirmative-action world. American Psychologist, 56(4), 302–318.