Variation and generality in encoding of syntactic anomaly information in sentence embeddings

Qinxuan Wu, Allyson Ettinger

TL;DR

This paper investigates how well NLP models encode information about syntactic anomalies in sentence embeddings. They found that while most models can detect specific anomalies, only recent transformer models like BERT and RoBERTa show signs of a generalized understanding of "sentence oddity," suggesting they learn more abstract error signals.

What Problem Does This Paper Address?

As NLP models become more sophisticated, understanding what linguistic knowledge they actually capture is crucial. While anomalies (like grammatical errors) have been used to test models, there's a lack of clarity on how models encode this anomaly information. Do they just learn specific patterns for each anomaly type, or do they develop a more general sense of what makes a sentence "wrong"? This paper aims to fill this gap by probing sentence embeddings for both specific anomaly detection and the generality of that detection ability.

Methods

The researchers designed four novel probing tasks, each focusing on a different type of syntactic anomaly:

  • Mod-Noun: Swapping modifiers and nouns (e.g., "a scarf yellow" instead of "a yellow scarf").
  • Verb-Ob: Swapping verbs and objects (e.g., "a bike rides" instead of "rides a bike").
  • SubN-ObN: Swapping subject and object nouns (e.g., "a bike rides a man" instead of "a man rides a bike").
  • Agree-Shift: Creating subject-verb number disagreement (e.g., "a man ride a bike" instead of "a man rides a bike").

Crucially, these tasks were designed to keep word content constant, isolating syntactic structure. They then tested six different sentence encoders (RNN-based and transformer-based) on their ability to distinguish between normal and anomalous sentences. To assess generality, they also performed transfer learning experiments, training a classifier on one anomaly type and testing it on another.

Datasets

The normal sentences used for creating the probing datasets were drawn from the MultiNLI (Natural Language Inference) dataset (premise-only splits). The perturbed sentences were generated by applying the four anomaly types described above to these MultiNLI sentences. For transfer learning experiments, they also incorporated established anomaly detection datasets: SOMO, BShift, CoordInv (from Conneau et al., 2018), and the CoLA (Corpus of Linguistic Acceptability) task (Warstadt et al., 2019).

Key Insights and Ideas

  • Hierarchy Matters: The hierarchical level at which an anomaly occurs significantly impacts detection. For instance, Verb-Ob anomalies (affecting a broader syntactic scope) were generally easier for models to detect than Mod-Noun anomalies.
  • World Knowledge is Tricky: The SubN-ObN task, which requires understanding argument structure and real-world plausibility (e.g., a bike can't ride a man), proved to be the most challenging for most models.
  • Transformers Lead the Pack: Modern transformer models (BERT, RoBERTa) significantly outperformed older RNN-based models across all anomaly types, especially on the more complex SubN-ObN task.
  • Generality is Rare: While most models can learn to detect specific anomalies, only BERT and RoBERTa showed evidence of a generalized "anomaly awareness" through successful transfer learning between different anomaly types. This suggests they might be learning a more abstract notion of sentence oddity.
  • Coarser Cues Can Help: Even when function words (like prepositions, articles) were removed, models still retained some ability to detect anomalies, indicating that coarser word order information can contribute to detection.

Key Results

  • Anomaly Detection Performance: BERT and RoBERTa achieved the highest anomaly detection accuracies across the board, with RoBERTa generally outperforming BERT. For example, on the Verb-Ob task, RoBERTa achieved 95.28% accuracy, while older models like InferSent struggled, often performing near chance (around 50%).
  • Transfer Learning: For most encoders, performance dropped significantly when tested on anomaly types they weren't trained on. However, BERT and RoBERTa showed non-trivial transfer performance, indicating a more generalized understanding. For instance, when trained on Mod-Noun, BERT and RoBERTa could still detect Verb-Ob anomalies with considerable accuracy.
  • Content Word Importance: Removing function words reduced anomaly detection accuracy for all models, but the drop was particularly pronounced for BERT and RoBERTa on the reordering tasks, highlighting their reliance on syntactic cues.

Limitations and Future Work

The authors acknowledge that their anomaly detection tasks might not perfectly disentangle syntactic sensitivity from general probability sensitivity. Models might simply be identifying anomalous sentences as less probable. Future work could explore this relationship further by examining model perplexities. Additionally, while transformer models show signs of generality, they still benefit from anomaly-specific cues, suggesting a combination of generalized and specific knowledge. The authors suggest further research into how training objectives influence the development of abstract anomaly awareness and how this awareness can be leveraged for improved model robustness.

Summary updated

Abstract

While sentence anomalies have been applied periodically for testing in NLP, we have yet to establish a picture of the precise status of anomaly information in representations from NLP models. In this paper we aim to fill two primary gaps, focusing on the domain of syntactic anomalies. First, we explore fine-grained differences in anomaly encoding by designing probing tasks that vary the hierarchical level at which anomalies occur in a sentence. Second, we test not only models' ability to detect a given anomaly, but also the generality of the detected anomaly signal, by examining transfer between distinct anomaly types. Results suggest that all models encode some information supporting anomaly detection, but detection performance varies between anomalies, and only representations from more recent transformer models show signs of generalized knowledge of anomalies. Follow-up analyses support the notion that these models pick up on a legitimate, general notion of sentence oddity, while coarser-grained word position information is likely also a contributor to the observed anomaly detection.
BibTeX
@inproceedings{wu2021variation,
  author = {Qinxuan Wu and Allyson Ettinger},
  title = {{Variation and generality in encoding of syntactic anomaly information in sentence embeddings}},
  booktitle = {BlackboxNLP @ EMNLP},
  year = {2021},
  doi = {10.18653/v1/2021.blackboxnlp-1.18},
}

Loading related content...

Figure