Abstract
As Large Language Models (LLMs) transition from simple chatbots to integrated workplace assistants, understanding their behavioral inclinations becomes crucial for effective deployment. This study applies the O*NET Interest Profiler to major LLMs—GPT-3.5, Gemini 1.5, ERNIE 3.5, and Spark V3.5—using Holland’s RIASEC framework. The results reveal that LLMs exhibit distinct "personalities" with a strong preference for Social and Artistic tasks. However, a significant "interest-competence mismatch" exists, as models perform best in Conventional, routine-based work. Furthermore, these interest profiles are sensitive to language and model updates, highlighting the impact of training data and alignment on AI behavior.
Introduction
Traditionally, LLMs are evaluated on performance metrics such as accuracy, speed, and reasoning. However, as they assume more significant roles in human-AI teams, understanding their "interests"—the intrinsic dispositions that drive satisfaction and performance in humans—is essential. This paper addresses the management challenge of deploying LLMs by mapping their behavioral inclinations. Do these models "prefer" certain types of work, and how do these preferences align with their actual capabilities?
Methodology
The researchers employed the Occupation Network’s (O*NET) Interest Profiler (OIP) short form, a 60-item psychometric instrument based on Holland’s RIASEC model (Realistic, Investigative, Artistic, Social, Enterprising, and Conventional).
The methodology involved:
- Prompting Strategy: LLMs were instructed to simulate human responses, rating 60 work tasks on a 5-point Likert scale.
- Statistical Rigor: To mitigate randomness, the authors used a General Linear Mixed Model (GLMM) with 20 replications per item.
- Competence Mapping: Models self-rated their competence, which was then validated against independent ratings from four human experts.
The study tested GPT-3.5, Gemini 1.5 Pro, ERNIE 3.5, and Spark V3.5, including various version snapshots to track evolutionary drift.
Key Findings and Results
The study identified several critical insights regarding the "personalities" of modern LLMs:
The Artistic/Social Bias and Interest Codes
Across the board, LLMs showed a marked preference for Social (helping/teaching) and Artistic (creating/designing) domains, while showing the least interest in Realistic (manual/physical) tasks. Each model displayed a unique "interest code" representing its top three domains.
The Interest-Competence Mismatch
A striking finding was the lack of correlation ($r = -0.060$) between a model's interest in a task and its actual competence. While LLMs "prefer" creative and social work, both self-ratings and expert evaluations confirmed they are most competent in the Conventional domain—organizing data and following routines.
Language and Evolutionary Drift
The "personality" of a model is not static. For instance, the ERNIE model showed a massive statistical shift ($P < 10^{-10}$) in interest scores when switched from English to Chinese, suggesting cultural nuances in training data deeply influence dispositions. Additionally, newer versions of GPT-3.5 showed shifted interest profiles, indicating that Reinforcement Learning from Human Feedback (RLHF) inadvertently reshapes model "personalities."
Discussion and Limitations
While the study provides a novel framework for AI evaluation, it faces certain limitations. The OIP is tailored to the U.S. labor market, and results might differ under other cultural frameworks. Furthermore, the "AI refusal" guardrails (where models claim to have no feelings) required specific prompting to bypass.
Future work should explore how these interests manifest in dynamic, long-form interactions and develop psychometric tools specifically designed for non-human agents. Understanding these traits is a prerequisite for effective human-AI collaboration and responsible role assignment in the future workforce.
Citation: Hua, M., Cheng, Y., & Zhu, H. (2024). The Career Interests of Large Language Models. arXiv:2407.08564v1 [cs.AI].
Summary updated