ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

Mike Zhang, Rob van der Goot, Barbara Plank

Introduction

The computational job market domain, encompassing tasks such as skill extraction from resumes and job title classification, is rapidly expanding. However, it faces a significant "data desert" challenge. While job advertisement data (JAD) is abundant, it is frequently proprietary, unstructured, and highly language-specific. Most existing Natural Language Processing (NLP) models are either general-purpose models like BERT or are restricted to a single language, lacking the nuance required for global professional terminology.

To address this, researchers introduced ESCOXLM-R, a multilingual language model optimized for the job market by leveraging the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy.

Methodology: Taxonomy-Driven Pre-training

ESCOXLM-R is built upon XLM-R_large, a robust multilingual transformer. The researchers employed Domain-Adaptive Pre-training (DAPT), further training the model on a specialized dataset rather than starting from scratch.

Overview of the ESCO taxonomy structure and the pre-training setup for the ESCO Relation Prediction (ERP) task.

The core innovation is a two-pronged training objective:

  1. Masked Language Modeling (MLM): A standard task where the model predicts missing tokens to learn general language patterns.
  2. ESCO Relation Prediction (ERP): A novel task where the model predicts the relationship between two professional concepts and their descriptions. Relationships are categorized as:
  3. Random: Unrelated concepts.
  4. Linked: Concepts appearing on the same ESCO occupation page (e.g., a skill required for a job).
  5. Grouped: Concepts belonging to the same high-level professional category.

This approach forces the model to internalize the hierarchical and relational structure of the professional world across 27 languages.

Datasets and Evaluation

The pre-training utilized the ESCO taxonomy, which contains approximately 3.72 million descriptions across 27 languages, covering over 13,000 skills and 3,000 occupations. The model was evaluated on nine JAD-related datasets across four languages: English (EN), Danish (DA), German (DE), and French (FR). These tasks included sequence labeling for entity extraction and multi-label classification.

Key Results and Insights

ESCOXLM-R achieved State-of-the-Art (SOTA) results on six of the nine evaluation datasets. Notable improvements included a gain of over 19 F1 points on certain English datasets and significant boosts in French (+10.3 F1) and German (+1.7 F1), demonstrating effective cross-lingual knowledge transfer.

Training loss and accuracy for the MLM and ERP objectives during the pre-training of ESCOXLM-R.

The model's success is attributed to several factors:

  • Structural Understanding: The ERP task allowed the model to associate skills with occupations across different languages.
  • Short-Span Specialization: ESCOXLM-R significantly outperformed general models on short spans (1–4 tokens), which is critical as most skills and job titles are brief.
  • Rare Entity Recognition: The model showed a superior ability to recognize a diverse range of unique, rare entities (the "long tail") rather than just memorizing common ones.
  • Efficiency: High-quality results were achieved after only ~1.04 epochs of training, highlighting the efficiency of using structured, human-curated data for domain adaptation.

Limitations and Future Directions

While successful, the model reflects the geographic focus of the ESCO taxonomy (primarily European), which may not capture all global nuances. Additionally, because ESCO lacks data on organizations and locations, the model did not improve performance on de-identification tasks. Future research may focus on creating "relation-centered" benchmarks to further test the model's understanding of the links between professional concepts.

Summary updated

Abstract

The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-R-large, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R-large on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.
BibTeX
@inproceedings{zhang2023escoxlm,
  author = {Mike Zhang and Rob van der Goot and Barbara Plank},
  title = {{ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.662},
}
Figure