Introduction
While models like BERT and ELMo have revolutionized Natural Language Processing (NLP), they are often treated as "black boxes." We know they work, but we don't fully understand how they store linguistic structures. Specifically, this research investigates hierarchical encoding within these models. In linguistics, sentences are nested hierarchies (constituency trees) rather than simple strings. This paper asks a granular question: To what extent is the hierarchical structure of a whole sentence compressed into the representation of a single word? If a single token's vector can "see" its ancestors in a parse tree, it suggests these models perform a sophisticated form of structural compression.
Methodology
The authors employ a technique called probing. Instead of fine-tuning BERT for a specific task, they "freeze" the model and train small, simple linear classifiers on top of the embeddings it produces.
- The Task: "Ancestor Sentiment Analysis." For any given word (a "leaf" in a tree), the probe must predict the sentiment (on a 5-class scale) of its parent node, its grandparent node, and the root node (the whole sentence).
- Layer-wise Probing: Every intermediate layer of BERT (base, cased) and ELMo variants were tested to observe how information evolves through the network.
- The Input: Crucially, the classifier only sees the representation of one word at a time, though that word has been contextualized by the model looking at the whole sentence.
- Baseline: Results were compared against GloVe, a non-contextual embedding, to prove that hierarchical knowledge stems from the contextualization process rather than the word's inherent meaning.
Datasets
The study utilizes the Stanford Sentiment Treebank (SST). This dataset provides sentiment labels for every constituent phrase in a sentence's parse tree, allowing researchers to map specific tokens to their hierarchical "ancestors" and their corresponding sentiment labels.
Key Results
The empirical results demonstrate a clear functional shift across the model's architecture. Lower layers (closer to the input) are optimized for local information, while higher layers (closer to the output) capture global, structural information.
As shown in the table above, BERT's performance varies significantly by layer:
- Leaf Sentiment: BERT’s first layer achieved its highest accuracy at 92.84%.
- Root Prediction: Using the representation of just one word from BERT’s 12th layer, a simple linear probe achieved 42.16% accuracy in predicting the sentiment of the entire sentence.
- Contextual Advantage: The non-contextual GloVe baseline managed only 28.81% for root prediction, confirming that the transformer architecture actively "bakes" sentence structure into individual token representations.
Discussion and Insights
- The "Holographic" Property: A single contextualized word vector acts like a microcosm of the entire sentence. It doesn't just represent a word; it represents that word's specific role and context within the broader hierarchical structure.
- The Depth Gradient: There is a distinct gradient where the optimal layer for prediction moves higher as the ancestor becomes more distant (from Leaf at Layer 1 to Root at Layer 12).
- Architecture Parity: Both LSTM-based (ELMo) and Transformer-based (BERT) architectures achieve comparable levels of hierarchical encoding when trained on similar data.
Conclusion and Future Work
This preliminary exploration confirms that contextualized embeddings store rich structural data. Future work could expand this to other probing tasks (such as tense or grammatical role) or explore whether larger models like RoBERTa or T5 encode hierarchy differently. While linear probes were used to ensure the information was truly "probed" rather than "learned," non-linear probes might reveal even deeper layers of encoded structural information.
Summary updated