Abstract
This study investigates the relationship between the internal weight structures of monolingual BERT models and the typological characteristics of the languages they represent. By introducing a novel weight-comparison metric called biCKA (Bidimensional Centered Kernel Alignment), we demonstrate that syntactic similarities between languages are directly reflected in the middle layers of their respective BERT models. This correlation becomes more pronounced when models are fine-tuned on parallel text, suggesting that semantic alignment clarifies the underlying structural linguistic rules encoded in the weights.
Introduction
Despite the massive success of Transformer models like BERT, their internal organization remains largely opaque. Traditional probing methods—training auxiliary models to predict linguistic features from frozen representations—measure what a model can do rather than how its physical architecture is organized. This paper addresses a more fundamental question: If two languages are structurally similar (e.g., sharing word order or morphological rules), do the weight matrices of their respective monolingual BERTs reflect this similarity? By comparing weights directly, we provide a "static" view of learned linguistic knowledge, independent of specific input sentences.
Methodology
Typological Framework and biCKA
The researchers utilized three primary components to bridge linguistics and deep learning. First, typological vectors were derived from the World Atlas of Language Structures (WALS) for 13 diverse languages, capturing features such as Subject-Verb-Object order and verb conjugation patterns. Second, the authors proposed biCKA (Bidimensional Centered Kernel Alignment), a novel variant of CKA designed to compare weight matrices directly. Unlike standard CKA, which compares activations, biCKA treats the weight matrix as a block diagonal matrix to account for relationships between both neurons (rows) and features (columns).
Experimental Setup
The study examined monolingual BERT-base models for 13 languages, including Romance (Italian, French, Spanish, Romanian), Germanic (English, Swedish, German, Dutch), Slavic (Russian), Hellenic (Greek), and non-Indo-European (Turkish, Finnish, Persian) families. To mitigate the "noise" of disparate pre-training data, a domain adaptation approach was employed by fine-tuning models on the EuroParl parallel corpus, providing 50,000 semantically equivalent sentences across the set.
Results and Discussion
Layer-wise Linguistic Correlation
The analysis revealed that syntactic properties are primarily handled in the "middle management" of the network. The strongest correlations between language structure and model weights appeared in layers 4–6. Specifically, the Value ($V$) matrix and the Attention Output ($OA$) dense layer showed the highest correlations, suggesting these components are the primary storage for structural linguistic rules.
In the full analysis, the correlation for the attention output matrix peaked at 0.66 in layer 5 ($OA_5$). Morphological similarity (how words are formed) showed a moderate correlation (approx. 0.55) but was localized to earlier layers (layer 3), indicating that morphology is processed lower in the stack than syntax.
Focused Analysis and Domain Adaptation
A focused analysis was conducted to identify "polarizing" features, such as the order of adjectives and nouns. By clustering languages based on these specific traits, the researchers found that structural differences strongly influenced weight similarity in layers 4 through 8.
Furthermore, domain adaptation through fine-tuning on parallel text significantly increased the correlation between weights and linguistic typologies, with an average boost of 0.18 in the middle layers. This suggests that when models are forced to represent the same meaning, the underlying structural similarities of their languages become much more visible in the weights.
Conclusion
This work provides a novel methodology for interpreting the internal structures of large language models by linking them to established linguistic typology. While limited by pre-training disparities and gaps in the WALS database, the results suggest that BERT models naturally organize linguistic information in a layer-wise hierarchy. This methodology could lead to "actionable explainability," allowing researchers to identify specific matrices for transfer learning or model editing based on a language's typological neighbors.
Summary updated