What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models

Wietse de Vries, Andreas van Cranenburgh, Malvina Nissim

Abstract

Peeking into the inner workings of BERT has shown that its layers resemble the classical NLP pipeline, with progressively more complex tasks being concentrated in later layers. To investigate to what extent these results also hold for a language other than English, we probe a Dutch BERT-based model and the multilingual BERT model for Dutch NLP tasks. In addition, through a deeper analysis of part-of-speech tagging, we show that also within a given task, information is spread over different parts of the network and the pipeline might not be as neat as it seems. Each layer has different specialisations, so that it may be more useful to combine information from different layers, instead of selecting a single one based on the best overall performance.

Introduction: The BERTology Pipeline

In the field of "BERTology," researchers previously discovered that English BERT seems to organize itself like a traditional software pipeline: it processes surface features (like spelling) in early layers, syntactic features (like sentence structure) in middle layers, and semantic features (like meaning and context) in the final layers.

However, it wasn't clear if this was a universal property of the Transformer architecture or just a quirk of English. This paper investigates two main questions:

  1. Does this "pipeline" behavior hold for a monolingual Dutch model (BERTje) and the multilingual BERT (mBERT)?
  2. Is the specialization of these layers actually "neat"? Does a layer that is good at Part-of-Speech (POS) tagging treat all words the same, or is the information spread out more chaotically than we thought?

Methodology: Probing the Black Box

The authors used probing classifiers (specifically the "edge probing" approach) to "peek under the hood" of the models without changing their weights. They extracted embeddings from every layer (0 through 12) and trained small, separate classifiers to perform specific linguistic tasks.

They employed two main probing techniques:

  • Scalar Mixing Probes: These learn a weighted average of all layers to identify which layers are most "informative" for a specific task.
  • Single Layer Probes: These test each layer in isolation to see how performance evolves step-by-step.

Crucially, they performed a fine-grained error analysis on POS tagging, separating "closed-class" words (like pronouns and conjunctions) from "open-class" words (like nouns and verbs) to see if the model processes them differently.

Datasets and Tasks

The study utilized several high-quality Dutch datasets to cover the spectrum of the NLP pipeline:

  • POS Tagging & Dependency Parsing: The UDLassy and UDAlpino datasets from Universal Dependencies v2.5.
  • Named Entity Recognition (NER): The Dutch portion of the CoNLL-2002 dataset.
  • Coreference Resolution: The SoNaR-1 corpus, identifying when different words refer to the same entity.

Key Insights: Universal Pipelines and Model Differences

The research confirms that both the Dutch-specific BERTje and the multilingual mBERT follow the same general "pipeline" order as the original English BERT. This suggests that the hierarchical processing of language is an inherent feature of the BERT architecture.

However, significant differences emerge between the models:

  • mBERT is "Faster": Interestingly, mBERT’s most informative layers appear earlier in the network than BERTje’s. For example, in POS tagging, mBERT peaks around layer 5, whereas BERTje peaks around layer 8. The authors hypothesize that because mBERT handles 104 languages with a shared vocabulary, it must start "generalizing" and resolving ambiguity sooner.
  • Lexical Reliance: BERTje relies much more heavily on its initial "lexical" layer (Layer 0) than mBERT does. This is likely because mBERT’s sub-word tokens (WordPieces) are shared across many languages, making them more ambiguous at the very start.
Weighted layer distributions across various NLP tasks for BERTje and mBERT.

Detailed Results: The Myth of the "Neat" Pipeline

While the overall performance for a task like POS tagging might peak at a specific layer, the model actually starts "forgetting" or transforming that information in later layers.

  • Accuracy Deltas: By measuring how much new information each layer adds, the authors found that while early layers provide massive gains, later layers often have "negative deltas," suggesting they might be discarding task-specific info in favor of general language modeling.
  • Open vs. Closed Class: F1 scores for closed-class words (like aux or det) remain stable once learned. However, for open-class words (like adj or noun), performance actually declines in the final layers. This proves that the "NLP pipeline" isn't just a series of additions; it's a series of transformations where some information is sacrificed to make room for others.

Conclusion

By showing that the "pipeline" is more of a messy, overlapping gradient than a series of clean steps, this paper provides a more nuanced map for researchers trying to interpret Transformer models. The findings suggest that instead of selecting a single "best" layer for a task, it may be more effective to combine information from different layers to capture the full range of linguistic specialization.

Summary updated

Abstract

Peeking into the inner workings of BERT has shown that its layers resemble the classical NLP pipeline, with progressively more complex tasks being concentrated in later layers. To investigate to what extent these results also hold for a language other than English, we probe a Dutch BERT-based model and the multilingual BERT model for Dutch NLP tasks. In addition, through a deeper analysis of part-of-speech tagging, we show that also within a given task, information is spread over different parts of the network and the pipeline might not be as neat as it seems. Each layer has different specialisations, so that it may be more useful to combine information from different layers, instead of selecting a single one based on the best overall performance.
BibTeX
@inproceedings{vries2020what,
  author = {Wietse de Vries and Andreas van Cranenburgh and Malvina Nissim},
  title = {{What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=pMYv4UL8PUz},
}
Figure