TL;DR
This paper introduces a novel method for generating "counterfactual embeddings" in neural language models, allowing researchers to causally probe how models use syntactic information. They found that while a masked language model (BERT) seems to leverage syntactic distance representations, a question-answering model (BERT fine-tuned for QA) does not, suggesting fine-tuning might obscure learned linguistic knowledge.
What Problem Does This Paper Address?
Neural language models like BERT and GPT-3 achieve impressive performance on various NLP tasks. However, understanding how they arrive at their predictions is a significant challenge. Researchers often use "probes" – trained models that map internal model representations (embeddings) to linguistic properties – to infer what information models encode. While probes can show that embeddings contain syntactic information, they don't definitively prove that the model uses this information for its predictions. This paper addresses this gap by developing a method to causally test whether models rely on the syntactic information detected by probes.
Methods
The core contribution is a technique for generating counterfactual embeddings. This method is inspired by causal analysis and uses trained syntactic probes to guide the manipulation of a model's internal embeddings. Specifically, they:
- Decompose the model: Split the language model (M) into an initial part (Mk-) that produces embeddings (zk) and a later part (Mk+) that uses these embeddings for output.
- Identify probe-relevant features (zp) and model-relevant features (zm): Define zp as the features of zk that a probe uses to predict syntax, and zm as the features of zk that Mk+ uses for its output.
- Generate counterfactual embeddings (z'): Using gradient descent, they iteratively modify the original embeddings (zk) to create new embeddings (z') that are designed to change the probe's output (i.e., predict a different syntactic parse). This is done by minimizing a loss function related to the probe's prediction.
- Causal intervention: They then feed these counterfactual embeddings (z') into the latter part of the model (Mk+) to observe how the model's final output changes. This allows them to test for a causal link between the syntactic information used by the probe (zp) and the information used by the model (zm).
This approach differs from prior work by actively manipulating embeddings to change probe outputs, rather than just observing correlations.
Datasets
The paper uses existing datasets for training and evaluation:
- Penn Treebank (PTB): Used for training the probes themselves.
- Coordination Corpus: A newly constructed corpus designed to test syntactic ambiguity related to pluralization. Sentences are structured to be interpretable with either a singular or plural verb, allowing researchers to see if the model's prediction of the masked word changes based on the intended parse.
- NP/Z Corpus: Another newly constructed corpus inspired by psycholinguistic studies of garden-path sentences. It features sentences with ambiguous adverbial or noun phrase attachment, allowing for testing of different parsing strategies.
- RC Corpus: Used for the question-answering model, featuring sentences with ambiguous relative clause attachment.
- NP/VP Corpus: Also used for the question-answering model, featuring sentences with ambiguous prepositional phrase attachment.
Key Insights and Ideas
- Causal probing is crucial: Simply observing that embeddings contain syntactic information (via probes) doesn't mean the model uses it. Counterfactual interventions are necessary to establish causality.
- Distance matters for masked language models: The masked language model (BERT) appears to leverage a "tree-distance-like" representation of syntax, as evidenced by distance-based probes successfully influencing its predictions.
- Fine-tuning might obscure knowledge: The BERT model fine-tuned for question-answering showed little to no consistent effect from counterfactual syntactic manipulations, suggesting that task-specific fine-tuning might lead models to rely less on or even discard some of the syntactic knowledge learned during pre-training.
- Probe complexity is important: Multi-layered, non-linear probes generally outperformed simpler linear probes, challenging the notion that simpler probes are always preferable for interpretability.
Key Results
- Masked Language Model (BERT):
- Distance-based probes (especially multi-layer ones) successfully shifted the Mask model's predictions for masked words according to intended syntactic parses in both the Coordination and NP/Z corpora. For example, generating counterfactuals based on an "adverb parse" increased the likelihood of adverb predictions, while a "noun parse" increased noun predictions.
- Depth-based probes showed much weaker and inconsistent effects.
- Question Answering Model (BERT fine-tuned for QA):
- Counterfactual embeddings generated using various probes (depth, distance, multi-layer distance) had minimal and inconsistent effects on the QA model's predictions across the RC and NP/VP corpora.
- The choice of layer for counterfactual insertion had a greater impact than the choice of parse, suggesting a lack of clear syntactic reasoning.
Limitations and Future Work
- Limitations:
- The paper acknowledges that their counterfactual generation method is a "gradient-based algorithm" and suggests that "more sophisticated counterfactual generation techniques" could yield more interesting results.
- While the results for the QA model suggest it may not use syntax, this is not definitively proven, as the authors state "admittedly cannot definitely prove."
- The study focused on syntactic probes; extending this to other linguistic properties is a natural next step.
- Future Work:
- Developing new probes that are informed by causal evidence.
- Exploring more complex counterfactual generation techniques.
- Investigating how fine-tuning affects the alignment between model and human representations of language.
- Further exploring the role of probe complexity and the trade-offs between different probe types.
Summary updated