Explainable AI in Language Learning: Linking Empirical Evidence and Theoretical Concepts in Proficiency and Readability Modeling of Portuguese

Luisa Ribeiro-Flucht, Xiaobin Chen, Detmar MeurersNAACL 2024

TL;DR

This paper tackles the challenge of understanding why AI models predict language learner proficiency and text readability, moving beyond simple accuracy. By applying Explainable AI (XAI) techniques to a comprehensive set of linguistic features in Portuguese, they validate theoretical assumptions and identify key linguistic markers that drive these predictions, offering valuable insights for language learning technology.

What Problem Does This Paper Address?

While AI has shown great promise in educational research, particularly in language learning, a significant hurdle remains: the "black box" problem. We often don't understand why an AI model makes a specific prediction. This lack of transparency hinders trust and limits our ability to refine models and gain deeper insights into the underlying linguistic phenomena. Specifically, this paper addresses the need for explainable models in assessing second language (L2) proficiency and text readability in Portuguese. Existing research often focuses on accuracy, but lacks detailed explanations of which linguistic features contribute most to these assessments, and whether these findings align with established linguistic theories. This is particularly important for languages other than English, where resources for such detailed analysis are scarcer.

Methods

The core of this research lies in its application of Explainable AI (XAI), specifically the Shapley Additive Explanations (SHAP) framework, to a set of linguistic complexity measures.

  • Linguistic Feature Extraction: The researchers utilized an extended version of the Common Text Analysis Platform (CTAP) to extract a comprehensive set of 465 linguistic complexity measures from Portuguese texts. These measures span five domains: superficial (e.g., counts, lengths), lexical (e.g., density, variation, sophistication), syntactic (e.g., clause complexity, phrase structure), morphological (e.g., inflections), and discourse-based (e.g., connectives, cohesion).
  • Classification Models: Standard supervised learning algorithms were employed, including Logistic Regression, Support Vector Machines (SVM), Random Forest, and a Multi-Layer Perceptron. SVM consistently showed the best performance.
  • Explainable AI (SHAP): The KernelExplainer method from the SHAP package was used to estimate the importance of each feature in making a prediction. SHAP values quantify the marginal contribution of each feature to the model's output, allowing researchers to understand whether a feature's presence or absence increases or decreases the predicted proficiency or readability level. This approach was chosen for its ability to provide both local (per-prediction) and global (overall) insights, regardless of model complexity.

Datasets

The study utilized two distinct corpora of Portuguese texts:

  • NLI-PT: This corpus consists of 3,069 L2 Portuguese learner texts, categorized into three general proficiency levels (A - Beginner, B - Intermediate, C - Advanced), aligned with CEFR levels.
  • Corpus de Complexidade Textual para Estágios Escolares do Sistema Educacional Brasileiro: This corpus comprises 2,076 Portuguese texts from Brazilian public school materials, categorized into four educational levels (Elementary School, Middle School, High School, University Education).

Both corpora exhibit class imbalance, which the authors acknowledge as a potential factor influencing results.

Key Insights and Ideas

  • Lexical Features Dominate: The most significant drivers for both proficiency and readability predictions were found to be lexical features. This strongly supports the theoretical emphasis on vocabulary richness, complexity, and familiarity in language acquisition and text comprehension.
  • Theoretical Validation: The XAI analysis largely corroborated existing linguistic theories and frameworks, such as the Common European Framework of Reference for Languages (CEFR) and the Portuguese Reference Level Descriptions (RLD). For instance, the acquisition of certain verb forms and syntactic structures at specific proficiency levels was reflected in the feature importance.
  • SVM's Robustness: The Support Vector Machine (SVM) classifier consistently outperformed other models, demonstrating its effectiveness in handling the complexity of linguistic features for both tasks.
  • Readability vs. Proficiency Nuances: While lexical features were key for both, there were some differences. For readability, imageability of word types emerged as a particularly strong predictor, alongside familiarity and age of acquisition. For proficiency, complex word frequency and specific vocabulary profile lists (A1, B1) were highly influential.
  • Syntactic and Morphological Contributions: While lexical features were paramount, syntactic features like clausal elaboration and morphological features like nominative case inflection also played significant roles, particularly in distinguishing between proficiency levels.

Key Results

  • Proficiency Classification: The best-performing SVM model achieved a state-of-the-art accuracy of 0.70 in the proficiency classification task (10-fold cross-validation) and 0.73 on the held-out test set.
  • Readability Classification: The SVM model achieved an accuracy of 0.84 for readability classification (10-fold cross-validation) and 0.85 on the held-out test set.
  • Feature Importance: SHAP analysis revealed that the top 20 discriminative features for proficiency included 10 lexical features, while for readability, 13 features were lexical. Complex word frequency, Portuguese Vocabulary Profile lists, and imageability were among the most impactful features.

Limitations and Future Work

  • Multicollinearity: The authors acknowledge that multicollinearity among highly correlated features can affect SHAP interpretation, potentially inflating or diminishing feature importance. While relative importance rankings remain informative, this is a consideration for precise interpretation.
  • Imbalanced Datasets: The class imbalance in both corpora could lead to an overemphasis on dominant class characteristics, potentially neglecting minority classes and affecting model performance.
  • Topic and Task Influence: The study did not account for the influence of distinct topics or tasks on linguistic features, which could affect grammar structures and lexical elements. Future work could explore corpus creation that considers these aspects.
  • Future Directions: The researchers aim to make their Portuguese analysis feature for CTAP openly accessible online, facilitating the integration of linguistic insights into educational technologies. They also suggest that future work could explore feature engineering and dimensionality reduction techniques while maintaining interpretability.

Summary updated

BibTeX
@inproceedings{ribeiroflucht2024explainable,
  author = {Luisa Ribeiro-Flucht and Xiaobin Chen and Detmar Meurers},
  title = {{Explainable AI in Language Learning: Linking Empirical Evidence and Theoretical Concepts in Proficiency and Readability Modeling of Portuguese}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure