TL;DR
This paper introduces a novel method for studying the inductive biases of neural language models by using carefully constructed artificial languages. They found that LSTMs are largely agnostic to word order, while Transformers show distinct preferences, though neither architecture's biases align with attested natural language tendencies.
What Problem Does This Paper Address?
Neural language models (like LSTMs and Transformers) are powerful but often act as "black boxes." It's challenging to understand why they perform well or poorly on certain tasks, especially when it comes to linguistic features. Specifically, researchers want to know if the model's architecture has an inherent "inductive bias" – a preference for certain linguistic structures or patterns. Investigating this with real-world languages is tricky because languages differ in countless ways (word order, morphology, syntax, etc.), making it hard to isolate the effect of a single feature. For example, comparing English (SVO) and Japanese (SOV) might show performance differences, but it's unclear if that's solely due to word order or other linguistic variations. This paper aims to solve this by creating a controlled environment to study these biases.
Methods
The core of this paper's contribution is its methodology: using artificial languages to isolate specific typological features. They construct simple, context-free grammars that are then modified by "switches." Each switch toggles the word order of specific grammatical constituents (e.g., subject-verb order, object-verb order). By systematically varying these switches, they generate multiple parallel corpora where sentences are identical except for the targeted typological variation. This allows for a fully controlled causal framework.
They then train both Transformer and LSTM language models on these artificial languages and measure their performance using perplexity. To analyze the results, they employ a linear mixed-effects model to quantify the effect of each switch and their interactions on model perplexity. This approach helps to disentangle the influence of individual features and their combinations.
Datasets
The paper does not use pre-existing natural language datasets for training and evaluation. Instead, they introduce a novel benchmark: artificial languages generated from context-free grammars. For each of the 64 possible grammar configurations (derived from 6 binary switches), they generate 100,000 sentences. These sentences are then split into train, development, and test sets (80-10-10) for each grammar. The generated corpora are designed to be parallel, meaning equivalent sentences are used across different grammars, differing only in the specific word order dictated by the switches.
Key Insights and Ideas
- Artificial Languages as a Controlled Lab: The paper convincingly argues that artificial languages are a powerful tool for dissecting inductive biases, offering a level of control unattainable with natural languages.
- Architectural Divergence: LSTMs and Transformers, despite being used for similar tasks, exhibit fundamentally different inductive biases regarding word order.
- LSTM Agnosticism: LSTMs appear remarkably indifferent to word order variations, showing similar performance across all tested grammatical configurations.
- Transformer Sensitivity: Transformers, in contrast, demonstrate clear preferences, with certain word order combinations leading to significantly better or worse performance.
- Mismatch with Nature: Crucially, neither architecture's biases seem to reflect the actual tendencies observed in attested natural languages, suggesting that current models might not be learning linguistic universals in the way humans do.
Key Results
- LSTMs: Showed very similar average perplexities across all 64 artificial grammars, indicating a lack of strong inductive bias towards any particular word order.
- Transformers: Exhibited significant variation in average perplexity across the grammars. Mixed-effects modeling revealed that individual switches (like S, VP, Comp, PP, NP, Rel) and their interactions had a substantial impact on transformer performance.
- Head-Directionality: For transformers, negative coefficients in the mixed-effects model for certain interaction terms (e.g., S and VP, S and Comp) suggest better performance when these constituents share head-directionality. However, this pattern doesn't hold universally and doesn't align with attested natural language tendencies.
- Rare Word Orders: The paper notes that the best performance for transformers was achieved on languages with the rarely-attested OVS sentence ordering, further highlighting the disconnect from natural language prevalence.
Limitations and Future Work
- Simplification of Grammars: The authors acknowledge that their artificial grammars are simple and do not encompass the full complexity of natural languages. Future work could involve extending these base grammars to test additional hypotheses.
- Limited Typological Scope: While their method allows for controlled variation, they note that their configurations cannot represent all possible word orders found in attested languages (e.g., VSO languages are not covered).
- Future Work: The authors suggest that their methodology can be extended to investigate other typological features beyond word order. They also mention that an automated method for assigning grammar weights could be explored. The current work focuses on a subset of typological variations, leaving room for broader investigations.
Summary updated