Longitudinal Complex Dynamics of Labour Markets Reveal Increasing Polarisation

Shahad Althobaiti, Ahmad Alabdulkareem, Judy Hanwen Shen, Iyad Rahwan, Morgan R. Frank, Esteban Moro, Alex Rutherford

Abstract

By analyzing seven decades of U.S. labor data using network science and natural language processing, this research demonstrates that the job market is becoming increasingly polarized. Occupations are diverging into two distinct silos—cognitive and physical—creating significant barriers to worker mobility and necessitating intensive retraining for transitions between these roles.

Introduction

Economic inequality is fundamentally linked to access to opportunity. As the structural map of available jobs fragments, social mobility stalls. This study investigates the long-term evolution of the labor market's "connective tissue"—the similarity of tasks between occupations—over a 70-year period to understand how automation and structural shifts affect worker transitions. While automation is often viewed as a sudden disruption, this paper takes a longitudinal view to identify underlying structural changes that have been unfolding for decades.

Methods and Datasets

The researchers utilized the Dictionary of Occupational Titles (DoT), covering five editions from 1939 to 1991, which they manually digitized. This was supplemented by modern O*NET data (1998–present) for benchmarking.

Summary statistics of the Dictionary of Occupational Titles (DoT) dataset across five editions.

The methodology combined several computational approaches:

  • Network Science: The labor market was represented as a graph where nodes are occupations and edges represent task similarity.
  • Natural Language Processing (NLP): Word embeddings (fastText) were used to analyze textual job descriptions for semantic similarity, moving beyond simple keyword matching to identify related roles like "teacher" and "instructor."
  • Machine Learning: State-of-the-art models (BERT, RoBERTa, and ELMo) were trained to classify roles as predominantly physical or cognitive.
  • Modularity Analysis: The Louvain method was applied to measure polarization, where high modularity indicates a network split into dense, isolated clusters with few bridging links.

Key Insights and Results

The analysis reveals a "Great Drift" where physical and cognitive jobs have moved further apart in task requirements. This polarization suggests a crumbling bridge between blue-collar and white-collar work.

Evolution of job similarity and title retention over time, highlighting the structural drift in the labor market.

Key findings include:

  • Steady-State Churn: Approximately 1.67% of job titles appear or disappear annually. This consistent turnover has been a feature of the US economy for 70 years, predating the modern AI era.
  • Structural Silos: Increased modularity indicates a less resilient network. Workers in physical roles hit by automation face structural barriers to transitioning into cognitive roles, necessitating more intensive retraining due to the lack of task overlap.
  • Classifier Performance: The ensemble model achieved 87% accuracy in distinguishing role types based on text alone. Validation against structured metadata confirmed that physical jobs have radically different interaction profiles with "Things" compared to cognitive jobs.
  • Long-term Trends: Polarization has increased steadily since 1939. While it stabilized somewhat after the 2008 Great Recession, 21st-century levels remain higher than the peaks seen in the 1990s.

Discussion and Limitations

The study focuses on the types of jobs available (structure) rather than employment volume. Additionally, it does not incorporate wage data, which could further illuminate the link between structural silos and pay stagnation. While the trend of polarization is clear, the specific causal drivers—technological change, offshoring, or regulatory shifts—remain a subject for future investigation.

Conclusion

The topography of the American workforce suggests a ladder of social mobility with increasingly missing rungs. The steady drift toward polarization indicates that the challenges of automation are not just about job loss, but about the increasing difficulty of navigating a fragmented labor landscape. Understanding these structural dynamics is crucial for developing effective retraining programs and labor policies that address the growing divide between physical and cognitive labor.

Summary updated

Abstract

In this paper we conduct a longitudinal analysis of the structure of labour markets in the US over 7 decades of technological, economic and policy change. We make use of network science, natural language processing and machine learning to uncover structural changes in the labour market over time. We find a steady rate of both disappearance of jobs and a shift in the required work tasks, despite much technological and economic change over this time period. Machine learning is used to classify jobs as being predominantly cognitive or physical based on the textual description of the workplace tasks. We also measure increasing polarisation between these two classes of jobs, linked by the similarity of tasks, over time that could constrain workers wishing to move to different jobs.
BibTeX
@misc{althobaiti2022longitudinal,
  author = {Shahad Althobaiti and Ahmad Alabdulkareem and Judy Hanwen Shen and Iyad Rahwan and Morgan R. Frank and Esteban Moro and Alex Rutherford},
  title = {{Longitudinal Complex Dynamics of Labour Markets Reveal Increasing Polarisation}},
  year = {2022},
  eprint = {2204.07073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07073},
}
Figure