CEFR-based Contextual Lexical Complexity Classifier in English and French

Desislava Aleksandrova, Vincent PouliotACL 2023

TL;DR

This paper introduces a novel approach to classifying lexical complexity in English and French, considering words and phrases within their context. By leveraging contextual embeddings from masked language models, the authors achieve improved accuracy in English and lay the groundwork for better French language learning tools, addressing a critical need for nuanced vocabulary assessment.

What Problem Does This Paper Address?

Accurately assessing the complexity of vocabulary is crucial for language learning applications, enabling personalized learning paths and effective content creation. Traditional methods often rely on decontextualized features like word frequency or length, which fail to capture the nuances of how a word's meaning and difficulty can change based on its usage. This is particularly challenging for polysemous words (words with multiple meanings) and multi-word expressions (MWEs). Existing lexical complexity classifiers often struggle with contextual understanding, especially for languages like French, where a lack of suitable contextualized datasets hinders development. The authors aim to bridge this gap by developing a CEFR-based classifier that considers context for both English and French, catering to the needs of the Mauril language learning platform.

Methods

The core of this work lies in developing a contextual lexical complexity classifier. The authors move beyond traditional feature-based approaches by employing contextual embeddings derived from pre-trained masked language models (MLMs), specifically BERT.

Here's a breakdown of their methodology:

  • Contextual Embeddings: Instead of static word embeddings (like Word2Vec), they use BERT (specifically bert-base-uncased for English and bert-base-multilingual-uncased for French) to generate dense vector representations for words and MWEs within their specific usage context. This allows the model to capture different meanings of polysemous words.
  • Vectorization Process: For each training example (a word/MWE, its context sentence, and its CEFR level), they encode the entire sentence using BERT. They then extract and sum the vector representations from all 12 hidden layers for each "WordPiece" (subword token) composing the target lexical item. Finally, these vectors are averaged to create a single representation for the item in context.
  • Classifier: A Support Vector Classifier (SVC) is used to predict the CEFR complexity level (ranging from A1 to C2). They employ adjusted class weights to handle class imbalances in the datasets.
  • Transfer Learning: To address the scarcity of French training data, they utilize a transfer learning approach. A multilingual BERT model is trained on English data and then applied to French, leveraging correlations between the two languages.

Datasets

The paper utilizes and introduces several datasets:

  • English Vocabulary Profile (EVP): A rich resource providing CEFR levels, part-of-speech tags, definitions, and usage examples for single words, phrasal verbs, phrases, and idioms in British and American English. This dataset was used for training and evaluating English models.
  • French Evaluation Corpus (FEC): Due to the lack of suitable contextualized French datasets, the authors created three versions:
  • FEC1: Extracted from "Les référentiels" (a French lexical reference), containing 914 entries with existing context examples.
  • FEC2: Also derived from "Les référentiels," but with usage examples created by a linguist to ensure consistency and avoid contradictions. It contains 473 lexical items.
  • FEC3: Based on Quebecois French textbooks, covering CEFR levels A1 to B2, with 48 lexical items per level. This corpus reflects competence rather than performance.
  • Mauril's pedagogical content: The ultimate goal is to support the Mauril platform, which uses a 4-class system (Beginner, Intermediate, Advanced, and a higher C class). The paper discusses how the CEFR levels map to Mauril's levels.

Key Insights and Ideas

  • Context is King (for Complexity): The paper strongly demonstrates that contextual information is not just helpful but essential for accurately predicting lexical complexity, especially for polysemous words. A word's meaning and thus its perceived difficulty can drastically change based on its surrounding text.
  • BERT's Power for Lexical Nuance: Masked language models like BERT are highly effective at capturing semantic nuances and word senses, making them superior to traditional frequency-based features for this task.
  • French Data Scarcity is a Major Hurdle: The significant drop in performance when evaluating on French data highlights the critical need for high-quality, contextualized French lexical resources. The authors' creation of the FEC datasets is a valuable contribution to address this.
  • Transfer Learning Shows Promise: The multilingual BERT approach demonstrates that knowledge learned from English data can be partially transferred to French, offering a viable strategy when direct French training data is limited.
  • Competence vs. Performance: The paper distinguishes between lexical items reflecting "competence" (what a learner should know) and "performance" (how words are actually used). The French datasets lean towards competence, which can differ from real-world usage.

Key Results

  • Improved English Performance: The contextualized BERT-based classifier (ME6 Contextual) significantly outperformed a baseline model using frequency and surface features, achieving F1 scores ranging from 0.42 to 0.63 across CEFR levels.
  • Moderate French Performance: The multilingual model (MME4 Contextual) showed moderate accuracy on French data, with F1 scores ranging from 0.05 to 0.70, but with a noticeable drop in intermediate classes.
  • Transfer Learning Benefits: The MMEFR4 Contextual model, trained on a combination of English and French data, showed improved F1 scores on French datasets (FEC2 and FEC3) compared to the purely English-trained multilingual model, without sacrificing English performance.
  • Class Imbalance Challenges: The paper notes that even with improvements, classification errors are often limited to neighboring CEFR classes, and the highest class (C2) remains challenging due to limited data.

Limitations and Future Work

  • French Data Quality: Despite efforts, the created French datasets (FEC) are acknowledged to have limitations, particularly in reflecting "performance" (actual usage) versus "competence." The authors note that the complexity of usage examples in FEC might not always perfectly align with the lexical item's complexity level.
  • Lack of Transparency: The dense, contextual embeddings make the classifier's predictions less interpretable, posing a challenge for understanding why a certain complexity level was assigned.
  • No Special Treatment for Cognates: The models do not explicitly handle cognates (words with shared origins), which are often easier for learners to acquire.
  • Human Validation Needed: The authors emphasize the need for human validation of the classifier's output, given the inherent complexities and limitations.

Future work could involve developing richer French datasets, exploring more interpretable models, incorporating cognate detection, and further refining the transfer learning approach.

Summary updated

BibTeX
@inproceedings{aleksandrova2023cefr,
  author = {Desislava Aleksandrova and Vincent Pouliot},
  title = {{CEFR-based Contextual Lexical Complexity Classifier in English and French}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure