On the Hierarchical Information in a Single Contextualised Word Representation (Student Abstract)

Dean L. Slack, Mariann Hardey, Noura Al Moubayed

Introduction

While models like BERT and ELMo have revolutionized Natural Language Processing (NLP), they are often treated as "black boxes." We know they work, but we don't fully understand how they store linguistic structures. Specifically, this research investigates hierarchical encoding within these models. In linguistics, sentences are nested hierarchies (constituency trees) rather than simple strings. This paper asks a granular question: To what extent is the hierarchical structure of a whole sentence compressed into the representation of a single word? If a single token's vector can "see" its ancestors in a parse tree, it suggests these models perform a sophisticated form of structural compression.

Methodology

The authors employ a technique called probing. Instead of fine-tuning BERT for a specific task, they "freeze" the model and train small, simple linear classifiers on top of the embeddings it produces.

  • The Task: "Ancestor Sentiment Analysis." For any given word (a "leaf" in a tree), the probe must predict the sentiment (on a 5-class scale) of its parent node, its grandparent node, and the root node (the whole sentence).
  • Layer-wise Probing: Every intermediate layer of BERT (base, cased) and ELMo variants were tested to observe how information evolves through the network.
  • The Input: Crucially, the classifier only sees the representation of one word at a time, though that word has been contextualized by the model looking at the whole sentence.
  • Baseline: Results were compared against GloVe, a non-contextual embedding, to prove that hierarchical knowledge stems from the contextualization process rather than the word's inherent meaning.

Datasets

The study utilizes the Stanford Sentiment Treebank (SST). This dataset provides sentiment labels for every constituent phrase in a sentence's parse tree, allowing researchers to map specific tokens to their hierarchical "ancestors" and their corresponding sentiment labels.

Key Results

The empirical results demonstrate a clear functional shift across the model's architecture. Lower layers (closer to the input) are optimized for local information, while higher layers (closer to the output) capture global, structural information.

Sentiment prediction accuracy across BERT layers for different hierarchical levels.

As shown in the table above, BERT's performance varies significantly by layer:

  • Leaf Sentiment: BERT’s first layer achieved its highest accuracy at 92.84%.
  • Root Prediction: Using the representation of just one word from BERT’s 12th layer, a simple linear probe achieved 42.16% accuracy in predicting the sentiment of the entire sentence.
  • Contextual Advantage: The non-contextual GloVe baseline managed only 28.81% for root prediction, confirming that the transformer architecture actively "bakes" sentence structure into individual token representations.

Discussion and Insights

  • The "Holographic" Property: A single contextualized word vector acts like a microcosm of the entire sentence. It doesn't just represent a word; it represents that word's specific role and context within the broader hierarchical structure.
  • The Depth Gradient: There is a distinct gradient where the optimal layer for prediction moves higher as the ancestor becomes more distant (from Leaf at Layer 1 to Root at Layer 12).
  • Architecture Parity: Both LSTM-based (ELMo) and Transformer-based (BERT) architectures achieve comparable levels of hierarchical encoding when trained on similar data.

Conclusion and Future Work

This preliminary exploration confirms that contextualized embeddings store rich structural data. Future work could expand this to other probing tasks (such as tense or grammatical role) or explore whether larger models like RoBERTa or T5 encode hierarchy differently. While linear probes were used to ensure the information was truly "probed" rather than "learned," non-linear probes might reveal even deeper layers of encoded structural information.

Summary updated

Abstract

Contextual word embeddings produced by neural language models, such as BERT or ELMo, have seen widespread application and performance gains across many Natural Language Processing tasks, suggesting rich linguistic features encoded in their representations. This work aims to investigate to what extent any linguistic hierarchical information is encoded into a single contextual embedding. Using labelled constituency trees, we train simple linear classifiers on top of single contextualised word representations for ancestor sentiment analysis tasks at multiple constituency levels of a sentence. To assess the presence of hierarchical information throughout the networks, the linear classifiers are trained using representations produced by each intermediate layer of BERT and ELMo variants. We show that with no fine-tuning, a single contextualised representation encodes enough syntactic and semantic sentence-level information to significantly outperform a non-contextual baseline for classifying 5-class sentiment of its ancestor constituents at multiple levels of the constituency tree. Additionally, we show that both LSTM and transformer architectures trained on similarly sized datasets achieve similar levels of performance on these tasks. Future work looks to expand the analysis to a wider range of NLP tasks and contextualisers.
BibTeX
@inproceedings{slack2020hierarchical,
  author = {Dean L. Slack and Mariann Hardey and Noura Al Moubayed},
  title = {{On the Hierarchical Information in a Single Contextualised Word Representation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7231},
}
Figure