Situational judgement tests: The why’s and how’s

Introduction: Why situation matters

IO Practitioners who want to make better talent decisions often use measures of ability and personality to inform their hiring practices. As we’ve discussed in previous posts, this is a very good idea indeed. Using objective, scientifically-validated assessments avoids subjective, biased decision-making and predicts job performance far beyond chance levels. However, it would be remiss not to also consider the situation within which work behavior occurs. Put another way, it is sometimes important to know how people would behave in very specific circumstances, rather than knowing their general disposition across situations.

To illustrate, imagine that a candidate, Joe, has a strong disposition to relate to others, be sociable, and build extensive interpersonal networks. We can surmise that Joe may be a good fit for a role that requires such competencies, such as a sales position. But what we don’t know is how Joe would react in a specific sales situation.

Although dispositions, values and motivations all affect behavior, we can measure a person’s likely reaction to situations separately with Situational Judgment Tests (SJTs).

SJTs: Advantages and pitfalls

SJTs have been used for a number of years to measure a person’s likely reactions (or judgements) given a specific situation. For instance, one might pose the following question in a sales-orientated SJT:

“A customer complains that the latest product advertised on the company’s website appears to be out-of-stock. What is your response?”

Questions like this one would often be followed by a number of potential responses, such as:

  1. Tell the customer that you can order the item they want
  2. Immediately inform your supervisor
  3. Apologize to the client and tell them that you’ll do your best to find an alternative product for them to consider
  4. Tell them of competing retailers that might stock the product

One of the key advantages of SJTs should be immediately apparent from the above example: their high face validity. For both candidates and hiring managers, the purpose of the questions is clear and uncontroversial.

Other benefits of SJTs include:

  • Applicability to the job in question. Specific scenarios ensure a close overlap between the requirements of the job and the applicant’s judgements
  • Because the items reveal typical situations that applicants are likely to face, SJTs communicate realistic employer expectations to potential hires
  • SJTs are ideal for high volume screening assessments where more expensive per-unit costs would make traditional testing prohibitively expensive
  • Robust predictive validity. When properly developed and applied, SJTs can add predictive power to selection assessments.

But SJTs are not without potential shortcomings. Much depends on the quality of item development as well as how closely the posited situations match actual working conditions.

Poorly constructed SJTs have a risk of telegraphing an obviously correct answer. In such cases, the SJT is actually measuring applicants’ ability to select the right option rather than measuring their likely judgements in such situations.

Fortunately, there are ways around this problem. At TTS, we prefer SJTs that requires test-takers to allocate points across different options, thus removing the notion of one, “correct” answer. This also ensures that we collect data on each statement which increases the overall reliability and validity of the instrument. In addition, possible response options should be constructed along a desirable gradient, thus further eliminating the temptation of finding the “right” answer.

Best-practice uses of SJTs

SJTs are not applicable to all selection processes. In fact, knowing when and where to employ SJTs is an important part of the IO Professional’s best practice toolkit. Research conducted by our assessment partner, cut-e, suggests some contexts where SJTs are best used:

  • High volume screening: Especially in jobs that require large numbers of candidates with similar skills
  • Semi-skilled, concrete roles: Advanced managerial and executive situations tend to be too complex to distill into easy-to-understand scenario descriptions
  • Competitive talent landscapes: In highly competitive employment markets, the speed and efficiency of SJTs can give recruiters a valuable advantage over competitors who use slower, more complicated selection methods
  • Unique environments: If a company operates in a domain that requires unique or niche behaviours, SJTs can quickly discern whether applicants have the right mix of motivations, values, and judgement that will help them succeed

Next steps

If you’re excited by the prospects of SJTs, you’re not alone. Research conducted by cut-e suggests that SJT use is on the rise throughout the testing world, with large companies being the most avid adopters of this method.

One caveat is that while the per-unit costs of an SJT is relatively low, developing the base items can be costlier. Therefore, it is useful to incorporate time and development costs into your implementation plan.

Despite this, SJTs offer a unique view of candidates and when combined with ability and personality-based tests, and add predictive power to an assessment battery. If you’re interested in how SJT’s can benefit your talent management process, why not drop us a line at info@tts-talent.com or visit our website at: www.tts-talent.com ?