TL;DR
This paper presents an approach using transformer adapters for predicting human eye-tracking behavior across multiple languages. By training language-specific adapters on frozen pre-trained language models, the team achieved second place in the CMCL 2022 shared task, demonstrating the effectiveness of this parameter-efficient method for understanding reading patterns.
What Problem Does This Paper Address?
Understanding how humans read is crucial for fields like cognitive science and natural language processing. Eye movements during reading, such as fixation durations and saccade lengths, offer rich insights into cognitive processes and how we process language. However, building models that can accurately predict these eye movements, especially across different languages, is challenging. Previous work has explored various methods, but the CMCL 2022 shared task specifically focused on multi- and cross-lingual prediction of eye-tracking features, pushing the boundaries of existing models to handle linguistic diversity and transfer learning.
Methods
The core of the proposed solution lies in transformer adapters. Instead of fine-tuning entire large pre-trained language models (PLMs) for each task or language, which can be computationally expensive and prone to catastrophic forgetting, the authors insert small, trainable "adapter" modules into the layers of frozen PLMs. These adapters learn task-specific or language-specific representations while the bulk of the PLM remains unchanged.
Specifically, they leverage the XLM-ROBERTa-base model, a multilingual PLM, and integrate adapters into its layers. They explore two main strategies:
- Single adapter for all languages: A single adapter is trained to handle eye-tracking prediction across all languages in the training set.
- Language-specific adapters: Separate adapters are trained for each language. This approach aims to overcome the "curse of multilinguality," where a single model might perform poorly on certain languages. They further explore a "stacked adapter" approach, where a pre-trained language adapter (e.g., from Wikipedia) is combined with a newly trained task-specific adapter.
For cross-lingual prediction (Subtask 2), they investigate zero-shot transfer and translation-based approaches, where the Danish test set is either directly used with a multilingual model or translated into a language seen during training (English or German).
Datasets
The primary datasets used are from the CMCL 2022 shared task. These datasets consist of eye-tracking corpora for six languages: English, Chinese, Russian, Hindi, German, and Dutch. The corpora vary in size and content (news articles, scientific texts, Wikipedia entries). For Subtask 2, a test set in Danish was provided, for which no training data was available.
Key Insights and Ideas
- Adapters are effective for eye-tracking prediction: The results demonstrate that parameter-efficient adapter modules can achieve performance comparable to, or even better than, full fine-tuning for predicting complex human behavior like reading.
- Language-specific adapters offer an advantage: Training separate adapters for each language significantly improved performance compared to a single adapter for all languages, highlighting the importance of capturing language-specific nuances in reading behavior.
- Stacked adapters show promise: Combining pre-trained language adapters with task-specific adapters yielded the best results, suggesting a synergistic effect.
- Translation for cross-lingual transfer works: Translating the test set into a closely related language present in the training data proved to be a viable strategy for cross-lingual transfer, especially when carefully selecting the target language (e.g., German for Danish, considering syntactic similarities).
- Wordpiece handling matters: Retaining all wordpieces generated by the tokenizer, rather than just the first one, led to better prediction accuracy, indicating that each sub-word unit can contribute to understanding reading behavior.
Key Results
The team achieved second place in the CMCL 2022 shared task leaderboard.
- Subtask 1 (Multi-lingual): The best performing model was the XLM-R setup with stacked language-specific adapters, achieving an MAE of 3.6533. This model outperformed the mean baseline and other adapter configurations.
- Subtask 2 (Cross-lingual): The best result for Danish prediction was achieved by translating the test set into German and using a trained adapter model, yielding an MAE of 4.9657. This approach outperformed zero-shot transfer and other translation strategies.
Limitations and Future Work
- Language-specific adapter availability: For some languages (like Dutch), pre-trained language-specific adapters were not readily available on AdapterHub, requiring the use of single, newly trained adapters.
- Performance variation across languages: The models performed better on some languages (German, Dutch) than others (Chinese, Russian), potentially due to differences in language typology, corpus characteristics, or vocabulary size.
- Large models: While exploring larger versions of XLM-ROBERTa, the benefits were not substantial, suggesting that adapters can effectively capture patterns without necessarily requiring larger base models, though further hyperparameter tuning might be beneficial.
Future work could explore:
- Dropping adapters or adapter fusion: Investigating more advanced adapter techniques.
- Autoregressive models: Considering different model architectures.
- Integrating other cognitive signals: Incorporating lexical, syntactic features, or other cognitive signals to enhance prediction.
- Inter-feature prediction: Using predictions from one eye-tracking feature to inform the prediction of another, as they represent different aspects of reading.
Summary updated