Abstract: Kabir Ahuja, Antonios Anastasopoulos, Barun Patra, Graham Neubig, Monojit Choudhury, Sandipan Dandapat, Sunayana Sitaram, Vishrav Chaudhary. Proceedings of the First Workshop on Scaling Up Multilingual Evaluation.
2022.
Introduction
Modern Multilingual Pre-trained Language Models (PLMs), such as XLM-R, are trained on over 100 languages. However, evaluation typically covers only a small fraction of these (e.g., the 15 languages in the XNLI benchmark). Zero-shot transfer—training in English and testing in a target language like Swahili—is often inconsistent, performing well for similar languages but failing for low-resource or typologically distant ones. Since creating gold-standard evaluation datasets for every language is prohibitively expensive, the SUMEval 2022 shared task challenged researchers to build regression models that predict PLM performance on unseen languages and tasks.
Methods and Features
The task was framed as a regression problem: given a tuple consisting of the target language, training configuration, pivot languages, model, and task, the goal was to predict a performance score (e.g., F1 or Accuracy).
The baseline LITMUS Predictor utilized XGBoost with four primary feature families:
- Pre-training Data Size: The volume of target language text in the model's original training set.
- Fine-tuning Data Size: The amount of labeled data used for task-specific training.
- Syntactic Distance: Linguistic divergence between the target and pivot languages (via the URIEL database).
- Sub-word Overlap: Shared tokens in the model's vocabulary between languages.
Top-performing teams introduced several innovations, including Joint Training (training a single model across multiple tasks to leverage cross-task transfer), New Features (such as writing scripts and binary flags for language presence in pre-training), and Ensemble Modeling (combining XGBoost, CatBoost, and MLPs).
Experimental Setup
The competition utilized performance data from XLM-Roberta and T-ULRv6 across four NLP tasks: XNLI (Natural Language Inference), TyDiQA (Question Answering), WikiANN (Named Entity Recognition), and UDPOS (Part-of-Speech Tagging). Evaluation was divided into two scenarios:
- Non-surprise: Predicting performance for languages seen in training but with new configurations.
- Surprise: Predicting performance for entirely new languages not present in the performance training set.
Key Results
The competition demonstrated significant improvements over the baseline. In the non-surprise variant, team PICT achieved a 64% reduction in Mean Absolute Error (MAE) compared to the LITMUS baseline. In the surprise variant, team GMU led with a 17% improvement in MAE.
The largest gains were observed in the TyDiQA task, where the baseline struggled due to data sparsity. Successful systems highlighted that multi-task learning and script-based features are critical for robust performance prediction.
Discussion and Limitations
While the results are promising, challenges remain. Prediction errors for certain languages like Amharic or Sindhi remained high (up to 0.25 F1-score), suggesting that current linguistic features may not fully capture the nuances of how PLMs process specific language families. Future work should explore scaling these predictors to a wider variety of architectures (e.g., GPT-style models) and more diverse, under-represented languages, potentially moving toward automated feature extraction directly from the models.
Summary updated