As large language models (LLMs) such as ChatGPT become increasingly accessible, many organizations are exploring how these tools might support key HR processes, especially in job and success profiling. For IO Psychologists this holds both promise and potential risks: Can AI help us do our work better, or does it risk diluting the very rigor that makes our work valuable?
At TTS, we believe the answer lies somewhere in the middle. A thoughtful, evidence-based approach to AI and success profiling will be one that respects the core principles and best practices of the IO Psychology discipline while embracing the opportunities offered by emerging technologies. In this article, we explore such an answer.
What makes success profiles successful?
A success profile is not just a list of desirable traits. It is a structured representation of the competencies, behaviors, knowledge, skills, and attributes that enable someone to succeed in a specific role within a specific context.
When developed well, a success profile serves as the bridge between real-world job demands and how to assess prospective talent for their likelihood of being suitable matches for the role.
To that end, a robust, well-developed success profile typically:
- Aligns with actual job performance criteria
- Enhances talent-job fit and retention
- Supports fairness and defensibility in selection
- Informs assessment choices across the talent lifecycle
The quality of a success profile depends heavily on how close it gets to the job itself. Getting close to the job has traditionally involved structured job analysis, SME input, and a deep understanding of the role’s organizational context.
However, with the advent of AI technologies like LLMs, other methods have become available.
The appeal and risks of using AI for job analysis
LLMs like ChatGPT offer appealing benefits to talent practitioners: Speed, accessibility, and the ability to generate fluent, structured outputs from a simple series of prompts.
Unsurprisingly, many HR teams have begun to utilize AI to produce job or success profiles, especially when time is at a premium. However, if used inappropriately or as a full substitute for professional analysis, this approach carries very real risks, such as:
- Loss of contextual nuance: AI cannot intuit the unique requirements of roles across different teams, business units, or organizational cultures.
- Validity concerns: Without real-world grounding, there is a danger of selecting irrelevant or poorly aligned competencies.
- “Frankenstein” profiles: When AI cobbles together generic content from broad knowledge bases, the result can be bloated, incoherent, or mismatched.
- Ethical and legal pitfalls: Profiles built without transparent, documented processes expose organizations to scrutiny, especially in high-stakes decision-making.
However, despite these risks, AI can enhance (not replace) the profiling process. How to navigate such complexities has been one of the major research concerns for our research team at TTS.
A closer look at ChatGPT and success profiling
To explore the real-world impact of using AI in success profiling, our research team conducted a multi-phase study comparing AI-generated and human-generated profiles across various roles.
A summary of their findings suggest the following:
- Moderate overlap between human and AI outputs: An average of 61% content agreement was found when comparing profiles generated by ChatGPT and those produced by SMEs.
- Improved reliability with better prompts: The use of more structured and detailed prompts significantly improved the inter-rater reliability of AI outputs, in some cases matching or exceeding that of human raters.
- Good to excellent inter-rater reliability: In phase two of the study, coefficients ranged from 0.759 to 0.899 within “groups” of different ChatGPT instances: A strong indicator of consistency, especially when improved prompts were used.
- Caution required:AI models occasionally added competencies not included in the source material (so-called AI “hallucinations”), underscoring the importance of human oversight.
These results confirm that AI can support success profiling processes when its outputs are carefully designed, constrained, and reviewed.
Principles for responsible AI use in success profiling
Based on our findings and broader IO Psychology practice, we propose five guiding principles for integrating AI into success profiling:
1. Use AI as a support tool, not a stand-in
AI can assist in structuring documents, generating initial drafts, or clarifying language: But it should never be the sole source of a success profile. Human insight remains central.
2. Anchor AI outputs in real job data
Use AI only in combination with:
- Role-specific information (job descriptions, performance goals)
- SME insights
- Organizational context and culture
The more grounded the input, the more useful the AI output.
3. Don’t skip work analysis
A valid success profile depends on thorough job analysis: Interviews, surveys, observations, or archival review. AI can assist with framing the results, but not with replacing the data collection process itself.
4. Develop prompt engineering skills
Prompt quality directly affects AI output quality. IO practitioners using LLMs should invest time in learning how to craft effective prompts, experimenting and testing different approaches to prompting, and understand how to interpret AI-generated content.
5. Maintain human oversight and ethical standards
IO Psychologists remain accountable for ensuring that profiling processes are fair, valid, and legally defensible.
Ethical concerns, including data privacy, transparency, and bias, must be explicitly addressed in any AI-assisted workflow.
A new role for IO Psychologists
The integration of AI into IO Psychology is not a threat: It is an invitation. As AI technologies mature, IO Psychologists can play a critical role in ensuring these tools are used appropriately, ethically and with scientific rigor.
Success profiling remains a core function of our discipline. AI may streamline aspects of this work, but only we, through our understanding of human behavior, organizational dynamics, and professional ethics can ensure it is done meaningfully.
Final thoughts
The future of success profiling is not about choosing between human or AI. It is about developing thoughtful processes in which human expertise and AI tools collaborate effectively.
Used well, AI tools like ChatGPT can improve consistency, clarity, and efficiency in profiling work. Used poorly or without understanding, it risks overly generic profiles, misalignment to organizational objectives, and loss of credibility. At TTS, we advocate for a balanced and integrated approach: One that keeps IO Psychology at the center of talent science, even as the tools around us evolve.
For more guidance on success profiling and how AI can be utilized within the discipline, why not connect with us at info@tts-talent.com?