Abstract
By analyzing seven decades of U.S. labor data using network science and natural language processing, this research demonstrates that the job market is becoming increasingly polarized. Occupations are diverging into two distinct silos—cognitive and physical—creating significant barriers to worker mobility and necessitating intensive retraining for transitions between these roles.
Introduction
Economic inequality is fundamentally linked to access to opportunity. As the structural map of available jobs fragments, social mobility stalls. This study investigates the long-term evolution of the labor market's "connective tissue"—the similarity of tasks between occupations—over a 70-year period to understand how automation and structural shifts affect worker transitions. While automation is often viewed as a sudden disruption, this paper takes a longitudinal view to identify underlying structural changes that have been unfolding for decades.
Methods and Datasets
The researchers utilized the Dictionary of Occupational Titles (DoT), covering five editions from 1939 to 1991, which they manually digitized. This was supplemented by modern O*NET data (1998–present) for benchmarking.
The methodology combined several computational approaches:
- Network Science: The labor market was represented as a graph where nodes are occupations and edges represent task similarity.
- Natural Language Processing (NLP): Word embeddings (fastText) were used to analyze textual job descriptions for semantic similarity, moving beyond simple keyword matching to identify related roles like "teacher" and "instructor."
- Machine Learning: State-of-the-art models (BERT, RoBERTa, and ELMo) were trained to classify roles as predominantly physical or cognitive.
- Modularity Analysis: The Louvain method was applied to measure polarization, where high modularity indicates a network split into dense, isolated clusters with few bridging links.
Key Insights and Results
The analysis reveals a "Great Drift" where physical and cognitive jobs have moved further apart in task requirements. This polarization suggests a crumbling bridge between blue-collar and white-collar work.
Key findings include:
- Steady-State Churn: Approximately 1.67% of job titles appear or disappear annually. This consistent turnover has been a feature of the US economy for 70 years, predating the modern AI era.
- Structural Silos: Increased modularity indicates a less resilient network. Workers in physical roles hit by automation face structural barriers to transitioning into cognitive roles, necessitating more intensive retraining due to the lack of task overlap.
- Classifier Performance: The ensemble model achieved 87% accuracy in distinguishing role types based on text alone. Validation against structured metadata confirmed that physical jobs have radically different interaction profiles with "Things" compared to cognitive jobs.
- Long-term Trends: Polarization has increased steadily since 1939. While it stabilized somewhat after the 2008 Great Recession, 21st-century levels remain higher than the peaks seen in the 1990s.
Discussion and Limitations
The study focuses on the types of jobs available (structure) rather than employment volume. Additionally, it does not incorporate wage data, which could further illuminate the link between structural silos and pay stagnation. While the trend of polarization is clear, the specific causal drivers—technological change, offshoring, or regulatory shifts—remain a subject for future investigation.
Conclusion
The topography of the American workforce suggests a ladder of social mobility with increasingly missing rungs. The steady drift toward polarization indicates that the challenges of automation are not just about job loss, but about the increasing difficulty of navigating a fragmented labor landscape. Understanding these structural dynamics is crucial for developing effective retraining programs and labor policies that address the growing divide between physical and cognitive labor.
Summary updated