TL;DR
This paper probes how BERT, a powerful language model, encodes linguistic features like word order and agreement across typologically diverse languages (English, Russian, Korean). By systematically testing BERT's layers on specific linguistic tasks, the authors reveal that while BERT generally captures these features, the depth at which they are encoded and the model's performance vary significantly across languages, offering insights into BERT's cross-lingual linguistic capabilities.
What Problem Does This Paper Address?
Pre-trained language models like BERT have revolutionized NLP, but they often remain "black boxes" from a linguistic perspective. While we have a general understanding that lower layers handle basic features and higher layers tackle more complex ones, there's a significant gap in systematically understanding how these models encode linguistic phenomena across languages with different structures. Specifically, this paper addresses the lack of systematic cross-linguistic comparisons of how BERT represents features like word order flexibility, morphological richness, and head directionality. Understanding these differences is crucial for improving BERT's performance in multilingual NLP tasks and for gaining deeper insights into the linguistic knowledge embedded within these models.
Methods
The authors employ a probing approach using BERT models (BERT-Base Cased for English, RuBERT for Russian, and BERT-Kor-base for Korean). They adapt the "bigram shift" (BShift) task, where a classifier is trained to detect if two adjacent tokens in a sentence have been swapped. This task is applied to different layers of the BERT model to assess how word order sensitivity evolves. For long-distance agreement, they use a masked token prediction task, evaluating the model's ability to predict the correct number (singular/plural) of a verb based on a distant noun. For gender bias, they mask pronouns, adjectives, and verbs and analyze the model's predictions for gendered forms. Crucially, they investigate the impact of tokenization strategies, particularly for morphologically rich languages like Korean.
Datasets
- UMC 0.1 Corpus: A Czech-Russian-English parallel corpus used for the word order corruption task.
- TED Talk Subtitles: A parallel corpus of Russian and Korean sentences used for the word order corruption task, specifically focusing on VO (Russian) and OV (Korean) structures.
- Gulordava et al. (2018) Dataset: A long-distance agreement test set for English and Russian, including both naturally occurring and syntactically correct but semantically nonsensical (nonce) sentences.
- Stanovsky et al. (2019) Dataset: Modified to create a test set for gender bias experiments, focusing on profession mentions and their associated pronouns, adjectives, and verbs.
Key Insights and Ideas
- Layered Linguistic Processing: BERT's layers do indeed correspond to a linguistic pipeline, with lower layers handling basic features and higher layers tackling more complex ones. However, the specific layers where optimal performance is achieved varies significantly by language and linguistic feature.
- Morphology Matters (A Lot): Rich morphology, particularly in Russian and Korean, plays a significant role in how linguistic features are encoded. Russian's rich inflectional morphology seems to help BERT encode word order more robustly at higher layers, while Korean's agglutinative morphology, when tokenized appropriately, allows for near-perfect word order detection even at the lowest layers.
- Tokenization is Key for Agglutinative Languages: For languages like Korean, where words are often split into stems and particles, morphology-aware tokenization is crucial for BERT to effectively learn word order. Whitespace tokenization hinders this process.
- Head Directionality is Less Critical Than Morphology: The study suggests that head directionality (VO vs. OV) has a less pronounced impact on word order detection compared to the language's morphological type.
- Long-Distance Agreement: Semantics vs. Syntax: While BERT shows a gap between performance on semantically plausible and nonce sentences for long-distance agreement, this gap diminishes at higher layers, indicating that BERT can learn syntactic relationships beyond simple co-occurrence.
- Capitalization's Subtle Role: For long-distance agreement, the cased version of BERT showed a slight advantage, suggesting capitalization might offer subtle cues for encoding morphological and syntactic relationships.
- Grammatical Gender vs. Semantic Bias: In Russian, the strong gender bias observed in BERT's predictions appears to be primarily driven by grammatical gender marking on nouns, verbs, and adjectives, rather than solely by semantic associations with professions, as seen in English.
Key Results
- Word Order: Russian models outperformed English models at middle and higher layers for word order corruption detection. Korean models with morphology-based tokenization achieved near-perfect accuracy (e.g., Layer 1: 0.988) on word order corruption, significantly outperforming other configurations.
- Long-Distance Agreement: BERT models showed a clear performance gap between original and nonce sentences, but this gap reduced at higher layers. Russian models generally performed better than English models, especially at the last layer.
- Gender Bias: Russian showed a strong skew towards masculine forms across pronouns, verbs, and adjectives, with winning rates of 93% for pronouns and 100% for verbs and adjectives. English pronouns were more evenly split (50% M, 50% F).
Limitations and Future Work
- Limited Languages: The study focused on only three languages. Future work should expand to more languages with diverse typological characteristics.
- Single Transformer Architecture: Only BERT-based models were analyzed. Further research should explore other Transformer architectures and training approaches.
- Task Specificity: The authors acknowledge that the performance on some tasks might be influenced by the specific dataset size and task constraints (e.g., restricted word order movements).
- Layer Optimization: The finding that optimal performance isn't always at the last layer suggests future work could explore using subsets of layers for specific downstream tasks.
- Investigating Other Morphological Types: The authors propose exploring languages like Spanish or German, which have rich morphology but differ from Russian, to see if adding more layers can improve performance.
Summary updated