Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages

Elena Sofia Ruzzetti, Federico Ranaldi, Felicia Logozzo, Michele Mastromattei, Leonardo Ranaldi, Fabio Massimo Zanzotto

Abstract

This study investigates the relationship between the internal weight structures of monolingual BERT models and the typological characteristics of the languages they represent. By introducing a novel weight-comparison metric called biCKA (Bidimensional Centered Kernel Alignment), we demonstrate that syntactic similarities between languages are directly reflected in the middle layers of their respective BERT models. This correlation becomes more pronounced when models are fine-tuned on parallel text, suggesting that semantic alignment clarifies the underlying structural linguistic rules encoded in the weights.

Introduction

Despite the massive success of Transformer models like BERT, their internal organization remains largely opaque. Traditional probing methods—training auxiliary models to predict linguistic features from frozen representations—measure what a model can do rather than how its physical architecture is organized. This paper addresses a more fundamental question: If two languages are structurally similar (e.g., sharing word order or morphological rules), do the weight matrices of their respective monolingual BERTs reflect this similarity? By comparing weights directly, we provide a "static" view of learned linguistic knowledge, independent of specific input sentences.

Methodology

Typological Framework and biCKA

The researchers utilized three primary components to bridge linguistics and deep learning. First, typological vectors were derived from the World Atlas of Language Structures (WALS) for 13 diverse languages, capturing features such as Subject-Verb-Object order and verb conjugation patterns. Second, the authors proposed biCKA (Bidimensional Centered Kernel Alignment), a novel variant of CKA designed to compare weight matrices directly. Unlike standard CKA, which compares activations, biCKA treats the weight matrix as a block diagonal matrix to account for relationships between both neurons (rows) and features (columns).

Experimental Setup

The study examined monolingual BERT-base models for 13 languages, including Romance (Italian, French, Spanish, Romanian), Germanic (English, Swedish, German, Dutch), Slavic (Russian), Hellenic (Greek), and non-Indo-European (Turkish, Finnish, Persian) families. To mitigate the "noise" of disparate pre-training data, a domain adaptation approach was employed by fine-tuning models on the EuroParl parallel corpus, providing 50,000 semantically equivalent sentences across the set.

Results and Discussion

Layer-wise Linguistic Correlation

The analysis revealed that syntactic properties are primarily handled in the "middle management" of the network. The strongest correlations between language structure and model weights appeared in layers 4–6. Specifically, the Value ($V$) matrix and the Attention Output ($OA$) dense layer showed the highest correlations, suggesting these components are the primary storage for structural linguistic rules.

Layer-wise Spearman correlation between syntactic/morphological similarities and biCKA weight similarity.

In the full analysis, the correlation for the attention output matrix peaked at 0.66 in layer 5 ($OA_5$). Morphological similarity (how words are formed) showed a moderate correlation (approx. 0.55) but was localized to earlier layers (layer 3), indicating that morphology is processed lower in the stack than syntax.

Focused Analysis and Domain Adaptation

A focused analysis was conducted to identify "polarizing" features, such as the order of adjectives and nouns. By clustering languages based on these specific traits, the researchers found that structural differences strongly influenced weight similarity in layers 4 through 8.

Syntactic similarities on focused clusters, highlighting the impact of specific polarizing linguistic features on model weights.

Furthermore, domain adaptation through fine-tuning on parallel text significantly increased the correlation between weights and linguistic typologies, with an average boost of 0.18 in the middle layers. This suggests that when models are forced to represent the same meaning, the underlying structural similarities of their languages become much more visible in the weights.

Conclusion

This work provides a novel methodology for interpreting the internal structures of large language models by linking them to established linguistic typology. While limited by pre-training disparities and gaps in the WALS database, the results suggest that BERT models naturally organize linguistic information in a layer-wise hierarchy. This methodology could lead to "actionable explainability," allowing researchers to identify specific matrices for transfer learning or model editing based on a language's typological neighbors.

Summary updated

Abstract

The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.
BibTeX
@inproceedings{ruzzetti2023exploring,
  author = {Elena Sofia Ruzzetti and Federico Ranaldi and Felicia Logozzo and Michele Mastromattei and Leonardo Ranaldi and Fabio Massimo Zanzotto},
  title = {{Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.963},
}
Figure