When IO Practitioners and talent professionals measure workplace behavior and ability they often turn to measures that determine a person’s standing on behavioral, personality, and cognitive constructs. As we have argued in previous articles, this is a powerful and scientifically defensible methodology that ensures high predictive validity for specific job fit.
Such assessments add value partly because they are cross-situational, in other words, they are valid across different contexts of work and are therefore not dependent on specific situations. They predict a person’s fit to the competencies required for the role across a variety of situations and contexts.
But there are also times when we are interested in how candidates will behave in very specific situations and contexts. Measuring such behavior is tricky and varies in fidelity. For instance, if we want to create a high-fidelity measure of specific context-dependent behavior, we may elect to let the candidate perform the actual work they are being assessed for. Such tests, known as work samples, give us insights into how a person may behave in the real world for a given job.
But work samples are not easy to construct, and in certain cases, may be almost impossible to devise given the complexities (or inherent risk and danger) of various roles. In such cases, simulations may be used, but they also tend to be time-consuming to develop and costly.
A practical and efficient alternative to such measures is the situational judgment test (SJT). Although SJTs are not as high in fidelity as work samples, they also target very specific work-related situations and decision-making.
In this article, we examine the basics of SJTs, their value, use, construction, and application.
What do SJTs measure?
As mentioned above, it is sometimes important to know how people would behave in very specific circumstances, rather than knowing their general disposition across situations.
To illustrate, imagine that a candidate, Sarah, has a strong disposition to relate to others, be sociable, and build extensive interpersonal networks. We can surmise that Sarah may be a good fit for a role that requires such competencies, such as a sales position. But what we don’t know is how she would react in a specific sales situation.
Although dispositions, values, and motivations all affect behavior, we can measure a person’s likely reaction to situations separately with SJTs.
In terms of predictive validity, meta-analytic studies have returned favorable results for SJTs. In one analysis conducted by McDaniel and his colleagues, SJTs’ criterion-related validity for job performance was around .26, which compares well with personality measures used in selection assessments.
How are SJTs constructed?
SJTs measure a person’s likely reactions (or judgments) given a specific situation. For instance, one might imagine the following situation in a sales environment:
A customer complains that the latest product advertised on the company’s website appears to be out-of-stock.
In an SJT, the candidate is asked to pick among a variety of likely responses to such a situation. For the above example, options may include:
- Tell the customer that you can order the item they want
- Immediately inform your supervisor
- Apologize to the client and suggest a similar product to the one advertised
- Tell them of competing retailers that might stock the product
To arrive at the appropriate situations and items requires thorough research and (often) consultation with clients about their specific work-related contexts and challenges.
Once the situations and options have been identified and constructed, validation research can be conducted to verify that the right behaviors are being assessed and that the weightings of options are appropriate for the specific situation or job.
Advantages of using SJTs
One of the key advantages of SJTs should be immediately apparent from the above example: high face validity. For both candidates and hiring managers, the purpose of the questions is clear and uncontroversial.
Another strength of SJTs is that in sketching likely real-world job situations, they can communicate realistic employer expectations to potential hires. SJTs are also ideal for high-volume screening assessments where more expensive per-unit costs would make traditional testing prohibitively expensive.
And because SJTs can be delivered online and unproctored, they represent a viable and efficient alternative to more labor-intensive methods like face-to-face assessment centers and traditional interviews.
Caveats when using SJTs
SJTs are not without potential shortcomings. Much depends on the quality of item development as well as how closely the posited situations match actual working conditions.
Poorly constructed SJTs have a risk of telegraphing an obviously correct answer. In such cases, the SJT is actually measuring applicants’ ability to select the right option rather than measuring their likely judgments in such situations.
Fortunately, there are ways around this problem. For instance, SJTs that require test-takers to allocate points across different options rather than selecting one, “correct” answer, counters this vulnerability. This also ensures that we collect data on each statement which increases the overall reliability and validity of the instrument.
In addition, possible response options should be constructed along a desirable gradient, thus further eliminating the temptation for test-takers to find the “right” answer.
Best practice applications of SJTs
Knowing when and where to employ SJTs is an important part of the IO and talent professionals’ best practice toolkit. Some applications where SJTs are best used include:
- High-volume screening. Especially in jobs that require large numbers of candidates with similar skills.
- Concrete roles. Advanced managerial and executive situations tend to be too complex to distill into easy-to-understand scenario descriptions, making SJTs ideal tests when applied to more concrete, practical roles.
- Competitive talent landscapes. In highly competitive employment markets, the speed and efficiency of SJTs can give recruiters a valuable advantage over competitors who use slower, more complicated recruitment and selection methodologies.
- Highly specific or unique work environments. If a company operates in a domain that requires unique or niche behaviors, SJTs can quickly discern whether applicants have the right mix of motivations, values, and judgment that will help them succeed.
- Skepticism of traditional psychometric measures. Despite their scientific validity and robust predictive power, psychometric tests sometimes fall out of favor with organizations, perhaps because they don’t always seem as obviously connected to real-world job tasks. In such cases, talent professionals and recruiters can turn to high face-validity SJTs as an alternative, yet still valid measure of job success.
Final thoughts
If you’re excited by the prospects of Situational Judgment Tests, you are not alone. Research suggests that SJT use is on the rise throughout the testing world, with large companies being the most avid new adopters of this method.
Situational Judgment Tests offer a unique view of candidates and when combined with ability and personality-based tests, can add predictive power to an assessment battery (not to mention considerable face validity).
If you’re interested in how Situational Judgment Tests can benefit your talent management process, or want to know about the different SJT products and solutions that TTS has to offer, why not drop us a line at info@tts-talent.com?
Sources
Whetzel, D.L., & McDaniel, M. (2009). Situational Judgment Tests: An overview of current research. Human Resource Management Review, 19, 188-202.