From Occupations to Tasks: A New Perspective on Automatability Prediction Using BERT

Dawei Xu, Haoran Yang, Marian-Andrei Rizoiu, Guandong Xu

TL;DR

This paper shifts the focus of automation research from broad "occupation-level" predictions to granular "task-level" analysis using BERT. By leveraging transformer-based NLP to understand the semantic nuances of task descriptions, the authors provide a more precise map of labor market vulnerability, estimating that roughly 25.1% of occupations face substantial automation risk.

What Problem Does This Paper Address?

For years, the "future of work" debate has been dominated by polarizing figures: some studies suggest nearly half of all jobs are at risk, while others put the number as low as 9%. The authors argue this discrepancy exists because most research treats occupations as monolithic blocks.

In reality, a single job consists of many diverse tasks. "Running a program" is highly automatable; "running a marathon" or "running a creative workshop" is not. Traditional models often miss these linguistic nuances. This paper addresses the need for a granular, task-based approach that uses modern NLP to distinguish between routine labor and tasks requiring human judgment, social intelligence, or complex physical dexterity.

Methods

The researchers developed a BERT-based classifier designed to categorize task statements into three levels of automation susceptibility:

  1. Substitution: Tasks that can be fully automated.
  2. Complementarity: Tasks that can be partially automated but still require human involvement.
  3. Negligibility: Tasks unlikely to be automated in the near future.

Key Technical Highlights: * Semantic Understanding: Unlike simpler models, BERT captures bidirectional context, allowing it to distinguish between different meanings of the same word based on the surrounding task description. * Data Augmentation via GPT-4: To combat class imbalance (where some categories had fewer examples), the authors used GPT-4 to paraphrase original task statements. This increased the training volume and improved the model's robustness to different wordings. * Expert Annotation: A panel of five experts used a voting mechanism to label the ground-truth data, ensuring the model was trained on high-quality, consensus-based assessments. * Attention Visualization: The authors used Wordclouds to visualize "Attention Weights," confirming that the model correctly focused on terms like "machinery operate" for substitution and "human expertise" for negligibility.

Datasets

The study utilized three major public datasets to ensure geographical and vocational diversity:

  • O*NET Task Statements: The primary database for US occupational characteristics.
  • ESCO Skills: The European classification of Skills, Competences, Qualifications, and Occupations.
  • Australian Labour Market Insights (AU LMI): Task data specific to the Australian workforce.

The researchers sampled over 13,000 tasks across these datasets for their analysis.

Key Insights and Ideas

  • The "Aha" Moment on Context: The paper demonstrates that the nature of the task is embedded in its semantics. Tasks involving "information providing" or "research conduct" correlate with recognized "automation bottlenecks" (social intelligence and complex problem solving), which the model successfully identified as low-risk.
  • Bridging the Gap: By aggregating task-level predictions back up to the occupation level, the authors arrived at a "middle-ground" estimate of 25.1% of occupations being at high risk. This provides a more nuanced and arguably more realistic figure than previous "all-or-nothing" occupational studies.
  • Sector-Specific Vulnerability: The model reveals a clear spectrum of risk. Industries like Accommodation and Food Services and Manufacturing are on the high-vulnerability end, while Education and Healthcare remain the most resilient due to the high level of human judgment and interaction required.

Key Results

  • Model Performance: The fine-tuned BERT model outperformed all baselines, including traditional ML (Random Forest, SVM) and other neural architectures (BiLSTM, GRU). On the combined dataset, BERT achieved a Precision of 0.7981, Recall of 0.7955, and an F1-score of 0.7944.
  • Robustness: The model proved remarkably stable; even when trained on only 20% of the data, it could match the performance of baseline models trained on 80% of the data.
  • Workforce Impact: The analysis of 974 ONET occupations found that 244 (25.1%) have a "substitution score" exceeding 50%. Conversely, 61.8% of occupations face a high risk of "complementarity," meaning AI will likely change how* the job is done rather than replacing the worker entirely.

Limitations and Future Work

  • Over-Augmentation Risks: The authors noted that while data augmentation helped, increasing the dataset by 4x or more led to a dramatic rise in precision that likely indicated overfitting. They recommend staying within the "Balanced" to "2 Times" augmentation range.
  • Bottleneck Evolution: While the paper identifies current bottlenecks (like fine motor skills), the authors acknowledge that technology moves fast. Future work could involve dynamic updates to the "bottleneck" definitions as robotics and AI evolve.
  • Policy Application: The authors suggest that the next step is using these granular insights to help policymakers design specific retraining programs for tasks that are most at risk, rather than trying to "save" entire disappearing occupations.

Summary updated

Abstract

As automation technologies continue to advance at an unprecedented rate, concerns about job displacement and the future of work have become increasingly prevalent. While existing research has primarily focused on the potential impact of automation at the occupation level, there has been a lack of investigation into the automatability of individual tasks. This paper addresses this gap by proposing a BERT-based classifier to predict the automatability of tasks in the forthcoming decade at a granular level leveraging the context and semantics information of tasks. We leverage three public datasets: O*NET Task Statements, ESCO Skills, and Australian Labour Market Insights Tasks, and perform expert annotation. Our BERT-based classifier, fine-tuned on our task statement data, demonstrates superior performance over traditional machine learning models, neural network architectures, and other transformer models. Our findings also indicate that approximately 25.1% of occupations within the O*NET database are at substantial risk of automation, with a diverse spectrum of automation vulnerability across sectors. This research provides a robust tool for assessing the future impact of automation on the labor market, offering valuable insights for policymakers, workers, and industry leaders in the face of rapid technological advancement.
BibTeX
@inproceedings{xu2023from,
  author = {Dawei Xu and Haoran Yang and Marian-Andrei Rizoiu and Guandong Xu},
  title = {{From Occupations to Tasks: A New Perspective on Automatability Prediction Using BERT}},
  booktitle = {BESC},
  year = {2023},
  doi = {10.1109/besc59560.2023.10386407},
}

Loading related content...

Figure