ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

Mike Zhang, Rob van der Goot, Barbara Plank

Introduction

The computational job market domain presents a unique set of challenges for Natural Language Processing (NLP). Job advertisements are characterized by domain-specific jargon, varying formats, and a significant "long tail" of rare skills and niche job titles. While general-purpose language models like BERT and XLM-R have revolutionized NLP, they often struggle with the nuances of recruitment data. Furthermore, most existing domain-specific models are monolingual, leaving a gap in multilingual support for tasks like skill extraction and job title classification.

To address these challenges, we introduce ESCOXLM-R, a multilingual language model specifically optimized for the job market. By leveraging the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, ESCOXLM-R bridges the gap between general language understanding and domain-specific expertise across 27 languages.

Methodology

ESCOXLM-R is built upon the XLM-R_large architecture and utilizes Domain-Adaptive Pre-training (DAPT). The core innovation lies in the introduction of a novel pre-training objective alongside standard Masked Language Modeling (MLM).

ESCO Relation Prediction (ERP)

The researchers introduced the ESCO Relation Prediction (ERP) task to force the model to internalize the hierarchical structure of the labor market. In this task, the model is presented with pairs of concepts and their definitions and must classify the relationship as:

  1. Random: Unrelated concepts.
  2. Linked: Concepts appearing on the same occupation page (e.g., "Animal Therapist" and "Animal Behavior").
  3. Grouped: Concepts belonging to the same broad category (e.g., different types of health professionals).

By treating the ESCO taxonomy as a knowledge graph, the model learns how skills relate to occupations and how different roles cluster together across multiple languages.

Experimental Setup

Pre-training Data

The model was pre-trained on the ESCO taxonomy, which contains approximately 3.72 million descriptions across 27 languages, covering over 3,000 occupations and 13,000 skills.

Evaluation Benchmarks

ESCOXLM-R was evaluated on 9 different datasets across 4 languages (English, Danish, German, and French), covering three primary task types:

  • Sequence Labeling (SL): Extracting specific text spans, such as skills or job titles.
  • Multilabel Classification (MLC): Assigning multiple skill tags to a single job description.
  • Multiclass Classification (MCC): Categorizing a skill into a specific taxonomy code.

Key Insights and Results

ESCOXLM-R achieved state-of-the-art (SOTA) results on 6 out of the 9 evaluation datasets. Several key findings emerged from the analysis:

  • Significant Performance Gains: In English sequence labeling tasks (SAYFULLINA and GREEN), the model improved F1 scores by over 19 points compared to previous benchmarks. It also showed strong cross-lingual performance, notably on the French FIJO dataset (+10.3 F1 points).
  • The "Short Span" Advantage: Job titles and skills are typically short (median length of ~3 tokens). ESCOXLM-R significantly outperforms general models on these short spans because it was pre-trained on concise taxonomy labels.
  • Handling the Long Tail: The model demonstrated superior Entity-F1 scores, indicating a better ability to recognize a diverse range of unique and rare entities that general models often overlook.
  • Structural Learning: The model achieved 60% accuracy on the ERP task, confirming that it successfully internalized the hierarchical logic of the ESCO taxonomy.

Limitations and Future Work

While ESCOXLM-R represents a significant step forward, it has limitations. The ESCO taxonomy is a European standard, which may introduce geographic bias when applied to non-Western job markets (as seen in the slightly lower performance on the Singaporean BHOLA dataset). Additionally, because ESCO focuses on skills and occupations, the model is less effective at de-identification tasks (e.g., the JOBSTACK dataset) that require recognizing organizations or personal names.

Future research will focus on detailed ablation studies to determine which specific sampling methods in the ERP task contribute most to performance and testing the model on more complex relation-centered tasks.

Summary updated

Abstract

The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-R, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.
BibTeX
@inproceedings{zhang2023escoxlm,
  author = {Mike Zhang and Rob van der Goot and Barbara Plank},
  title = {{ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.662},
}
Figure