TL;DR
This paper investigates "parameter-free probing" as a way to understand what linguistic structures Large Language Models (LLMs) like BERT learn internally, without relying on human-defined labels. The study finds significant discrepancies between BERT's learned structures and the Universal Dependencies (UD) linguistic formalism, suggesting that our current linguistic theories might not be the best analogy for interpreting LLMs.
What Problem Does This Paper Address?
Probing LLMs is a common technique to understand what linguistic knowledge they encode. Traditional probing methods involve training a separate classifier (a "probe") to predict human-annotated linguistic labels (like part-of-speech tags or dependency relations) from the LLM's internal representations. However, critics argue that this approach is biased because it imposes pre-defined, theory-laden labels onto the LLM, potentially telling us more about the probe than the LLM itself.
To overcome this, "parameter-free probing" methods have emerged. These techniques aim to extract structural information directly from the LLM without any additional training or external labels. The idea is that this "bottom-up" approach reveals the LLM's "intrinsic grammar." This paper specifically examines a parameter-free technique called "perturbed masking" applied to BERT, aiming to understand if it truly uncovers a genuine internal grammar and how this relates to established linguistic formalisms like Universal Dependencies (UD).
Methods
The core method used is perturbed masking, as introduced by Wu et al. (2020). This technique works by:
- Masking a token: For a given token $x_i$ in a sentence, its representation is obtained from BERT.
- Masking a pair of tokens: Then, both $x_i$ and another token $x_j$ are masked.
- Measuring impact: The difference (Euclidean distance) between the representation of $x_i$ when it's masked alone versus when both $x_i$ and $x_j$ are masked is calculated. This difference quantifies the "impact" of $x_j$ on $x_i$.
- Constructing a graph: These impact scores between all token pairs form an "impact matrix." This matrix is then used by the Eisner algorithm to construct a directed dependency graph, where the intuitive idea is that heads have the highest impact on their dependents.
The paper then compares the dependency graphs generated by BERT using this method against the Universal Dependencies (UD) annotation for English. To do this, they compare the dependency relation types (deprels) assigned by UD to the heads assigned by BERT.
Datasets
The study uses the English Parallel Universal Dependencies (PUD) dataset. This dataset consists of 1000 sentences, of which 7 were discarded due to specific issues with token ID mismatches between UD and BERT parses in coordinated sentences. The remaining data covers 21,047 UD-annotated tokens.
Key Insights and Ideas
- BERT's "Root" Attraction: A striking finding is BERT's strong tendency to assign the "root" (typically the main verb in UD) as the head for many tokens, even across phrase boundaries and for embedded clauses. This "attraction by the root" is not a common feature in traditional linguistic analysis.
- Lack of Recursion: BERT's dependency structures show a significant lack of recursion, particularly in embedded clauses. It often assigns arguments of embedded clauses to the main clause verb (the root), deviating from UD and standard linguistic theories that treat embedded structures distinctly.
- Flipped Headedness in PPs: Prepositional phrases (PPs) exhibit a systematic flip in headedness. While UD typically treats the preposition as a dependent of the noun phrase it modifies, BERT often treats the PP complement as the dependent of the preposition.
- Challenging Linguistic Analogies: The observed discrepancies are so fundamental that they question the assumption that LLMs like BERT represent grammar in a way that aligns with familiar linguistic formalisms. This suggests that we might need to develop new ways to interpret LLM outputs, rather than forcing them into existing linguistic boxes.
Key Results
- 58% Shift: Approximately 58% of all tokens showed a shift in their head-dependent relationship between UD and BERT parses.
- Root as Head: BERT assigned the root as a head in 35% of all shifts, a significantly higher proportion than UD.
- Embedded Clause Deviations: For embedded clauses, BERT's shift ratios ranged from 64% to 94%, indicating a substantial departure from UD.
- PP Headedness Flip: The paper highlights a consistent reversal of headedness in prepositional phrases between BERT and UD.
- Local Dependencies Not the Whole Story: While Wu et al. (2020) suggested BERT learns local dependencies, the study found that the average dependent-head distance in BERT parses was actually higher than in UD, suggesting locality alone doesn't explain the differences.
Limitations and Future Work
Limitations:
- Single Model, Technique, and Dataset: The study is limited to one LLM architecture (BERT), one parameter-free probing technique (perturbed masking), and one English dataset (PUD).
- Manual Analysis: While quantitative data is presented, detailed qualitative analysis of specific parse-pairs was manual and thus has limited coverage.
- Token ID Discrepancies: A minor issue with token ID mismatches in coordinated sentences led to the exclusion of a small portion of the data.
Future Work:
- Inter-lingual Comparison: Extending the analysis to multiple languages is crucial for understanding cross-lingual transfer and universal linguistic properties in LLMs.
- Multiple Probing Techniques: Investigating other parameter-free probing methods would provide a more comprehensive view.
- Alternative Formalisms: Exploring how BERT's structures align with other linguistic formalisms, such as Surface-Syntactic Universal Dependencies (SUD), could be fruitful, as BERT's behavior might be more compatible with them.
- Developing LLM-centric Interpretations: The paper strongly advocates for developing new ways to interpret LLMs that are not solely reliant on pre-existing linguistic theories, suggesting a move towards explaining LLMs "in their own terms."
Summary updated