TL;DR
This paper introduces a novel, data-driven framework for assessing AI's impact on jobs by leveraging Large Language Models (LLMs) to evaluate the AI suitability of specific job tasks. The authors develop two key indices, TEAI (Task Exposure to AI) and TRAI (Task Replacement by AI), and find that AI exposure is positively correlated with cognitive and problem-solving skills, and surprisingly, with employment and wage growth, suggesting a complementary rather than purely substitutive role for AI in the current labor market.
What Problem Does This Paper Address?
Estimating the extent to which AI technologies will automate or augment human jobs is a critical challenge for policymakers, businesses, and individuals. Previous approaches often relied on external benchmarks like expert opinions or patent data, which can be subjective or fail to capture the rapid advancements in AI, particularly in areas like natural language processing. This paper tackles this by proposing a reproducible, internal assessment method that uses LLMs themselves to evaluate AI's capability to perform specific job tasks, offering a more dynamic and granular understanding of AI's impact across the US labor market.
Methods
The core of the paper is a three-step framework:
- Compute AI Rate (TEAI): The authors employ three leading open-source LLMs (Mistral 7B Instruct, OpenChat 3.5, and Orca Mini v3) to assess the suitability of over 19,000 ONET tasks for AI performance on a scale of 1-5. This involves a five-shot prompt design, asking LLMs to consider AI technologies like LLMs, image processing, and robotics. To mitigate LLM "hallucinations," a consensus mechanism is used to derive a single AI rate (TE) for each task. This TE score is then aggregated at the occupation level, weighted by task relevance, importance, and frequency from ONET, to create the TEAI (Task Exposure to AI) index.
- Compute AI Substitutability (TRAI): Leveraging the natural language motivations provided by LLMs in the first step, the authors synthesize these justifications. This summarized motivation, along with the task and occupation title, is fed into a larger LLM (Qwen 72B) to generate a "task replacement" score (1-5), indicating the degree to which AI can automate or replace human execution of that task. This is then aggregated to create the TRAI (Task Replacement by AI) index.
- Human Evaluation: To validate their LLM-generated indices, the authors conducted a user study with 12 evaluators who assessed the TEAI and TRAI scores and motivations for a sample of occupations and tasks.
Datasets
- O*NET Database: The primary dataset used is the O*NET taxonomy, which provides detailed information on over 1,000 occupations and more than 19,000 job-related tasks. This serves as the reference for defining job tasks and occupations.
- US BLS Employment Data: Used to analyze the distribution of AI exposure across the US population and different occupational and skill groups.
- ACS (American Community Survey) Data: Utilized for analyzing the relationship between AI exposure and worker characteristics like education and age.
Key Insights and Ideas
- LLMs as AI Assessors: The paper innovatively uses LLMs to self-assess AI capabilities, moving beyond traditional external benchmarks. This approach is inherently reproducible and adaptable to new LLM generations.
- Complementarity Over Substitution: Contrary to some fears, the study suggests AI is more likely to complement human labor, especially in high-skill jobs, leading to increased productivity and wage growth, rather than outright replacement.
- Skill Correlation: AI exposure is strongly correlated with cognitive, problem-solving, and management skills, but negatively with social skills, highlighting AI's current strengths and weaknesses.
- High-Skill Job Impact: A significant portion of the US workforce, particularly those in high-skill jobs requiring higher education, is highly exposed to AI.
- Dynamic Nature of AI Impact: The framework's ability to incorporate new LLMs allows for tracking the evolving impact of AI over time.
Key Results
- TEAI and Skills: TEAI is positively correlated with cognitive, problem-solving, and managerial skills, and negatively with social skills.
- AI Exposure Distribution: In 2023, 34% of US employment was highly exposed to AI, with 88% of this highly exposed group being in high-skill jobs.
- Employment and Wage Growth: AI exposure is positively correlated with employment and wage growth over the 2003-2023 period, suggesting a net positive impact on productivity.
- Human Evaluation Agreement: Evaluators agreed with the TEAI and TRAI indices and their explanations 71-75% of the time, indicating the indices are meaningful.
- TEAI vs. TRAI: The TEAI and TRAI indices are highly correlated, but TRAI specifically highlights the potential for task substitution, even within high-skill occupations, suggesting a shift in task allocation rather than complete job replacement.
Limitations and Future Work
- O*NET Taxonomy Static Nature: The O*NET taxonomy, while comprehensive, is static and may not capture emerging tasks or entirely new job roles. The authors suggest their framework is adaptable to user-defined taxonomies to address this.
- LLM Limitations: While efforts were made to mitigate LLM "hallucinations" through consensus, the inherent limitations of LLMs remain a factor.
- Future Work: The authors emphasize the reproducibility of their work and encourage the community to use their framework to track AI's progress and inform policy decisions for AI integration into the workforce. They also highlight the need to understand the nuances of complementarity versus substitution in AI's impact on specific tasks and occupations.
Summary updated