The Career Interests of Large Language Models

Meng Hua, Yuan Cheng, Hengshu ZhuarXiv 2024 · DOI · OpenReview

Abstract

As Large Language Models (LLMs) transition from simple chatbots to integrated workplace assistants, understanding their behavioral inclinations becomes crucial for effective deployment. This study applies the O*NET Interest Profiler to major LLMs—GPT-3.5, Gemini 1.5, ERNIE 3.5, and Spark V3.5—using Holland’s RIASEC framework. The results reveal that LLMs exhibit distinct "personalities" with a strong preference for Social and Artistic tasks. However, a significant "interest-competence mismatch" exists, as models perform best in Conventional, routine-based work. Furthermore, these interest profiles are sensitive to language and model updates, highlighting the impact of training data and alignment on AI behavior.

Introduction

Traditionally, LLMs are evaluated on performance metrics such as accuracy, speed, and reasoning. However, as they assume more significant roles in human-AI teams, understanding their "interests"—the intrinsic dispositions that drive satisfaction and performance in humans—is essential. This paper addresses the management challenge of deploying LLMs by mapping their behavioral inclinations. Do these models "prefer" certain types of work, and how do these preferences align with their actual capabilities?

Methodology

The researchers employed the Occupation Network’s (O*NET) Interest Profiler (OIP) short form, a 60-item psychometric instrument based on Holland’s RIASEC model (Realistic, Investigative, Artistic, Social, Enterprising, and Conventional).

The experimental workflow: from O*NET tasks to LLM interest profiling.

The methodology involved:

  1. Prompting Strategy: LLMs were instructed to simulate human responses, rating 60 work tasks on a 5-point Likert scale.
  2. Statistical Rigor: To mitigate randomness, the authors used a General Linear Mixed Model (GLMM) with 20 replications per item.
  3. Competence Mapping: Models self-rated their competence, which was then validated against independent ratings from four human experts.

The study tested GPT-3.5, Gemini 1.5 Pro, ERNIE 3.5, and Spark V3.5, including various version snapshots to track evolutionary drift.

Key Findings and Results

The study identified several critical insights regarding the "personalities" of modern LLMs:

The Artistic/Social Bias and Interest Codes

Across the board, LLMs showed a marked preference for Social (helping/teaching) and Artistic (creating/designing) domains, while showing the least interest in Realistic (manual/physical) tasks. Each model displayed a unique "interest code" representing its top three domains.

LLM interest codes and corresponding human occupations.

The Interest-Competence Mismatch

A striking finding was the lack of correlation ($r = -0.060$) between a model's interest in a task and its actual competence. While LLMs "prefer" creative and social work, both self-ratings and expert evaluations confirmed they are most competent in the Conventional domain—organizing data and following routines.

Language and Evolutionary Drift

The "personality" of a model is not static. For instance, the ERNIE model showed a massive statistical shift ($P < 10^{-10}$) in interest scores when switched from English to Chinese, suggesting cultural nuances in training data deeply influence dispositions. Additionally, newer versions of GPT-3.5 showed shifted interest profiles, indicating that Reinforcement Learning from Human Feedback (RLHF) inadvertently reshapes model "personalities."

Discussion and Limitations

While the study provides a novel framework for AI evaluation, it faces certain limitations. The OIP is tailored to the U.S. labor market, and results might differ under other cultural frameworks. Furthermore, the "AI refusal" guardrails (where models claim to have no feelings) required specific prompting to bypass.

Future work should explore how these interests manifest in dynamic, long-form interactions and develop psychometric tools specifically designed for non-human agents. Understanding these traits is a prerequisite for effective human-AI collaboration and responsible role assignment in the future workforce.

Citation: Hua, M., Cheng, Y., & Zhu, H. (2024). The Career Interests of Large Language Models. arXiv:2407.08564v1 [cs.AI].

Summary updated

Abstract

Recent advancements in Large Language Models (LLMs) have significantly extended their capabilities, evolving from basic text generation to complex, human-like interactions. In light of the possibilities that LLMs could assume significant workplace responsibilities, it becomes imminently necessary to explore LLMs' capacities as professional assistants. This study focuses on the aspect of career interests by applying the Occupation Network's Interest Profiler short form to LLMs as if they were human participants and investigates their hypothetical career interests and competence, examining how these vary with language changes and model advancements. We analyzed the answers using a general linear mixed model approach and found distinct career interest inclinations among LLMs, particularly towards the social and artistic domains. Interestingly, these preferences did not align with the occupations where LLMs exhibited higher competence. This novel approach of using psychometric instruments and sophisticated statistical tools on LLMs unveils fresh perspectives on their integration into professional environments, highlighting human-like tendencies and promoting a reevaluation of LLMs' self-perception and competency alignment in the workforce.
BibTeX
@misc{hua2024career,
  author = {Meng Hua and Yuan Cheng and Hengshu Zhu},
  title = {{The Career Interests of Large Language Models}},
  year = {2024},
  eprint = {2407.08564},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.08564},
}
Figure