Perturbed examples reveal invariances shared by language models

Ruchit Rawal, Mariya TonevaSubmission to ICLR 2024 · Rejected

TL;DR

As standard NLP benchmarks reach saturation, simple accuracy scores fail to capture how models truly differ. This paper introduces SCoPE, a framework that measures "shared invariances"—how similarly two models react to specific linguistic changes like synonyms or typos. It reveals that larger models possess a more diverse set of behavioral patterns that smaller models simply haven't "unlocked" yet.

What Problem Does This Paper Address?

We are currently in an era of "benchmark saturation." When multiple models all score 90%+ on a task like sentiment analysis, how do we know which one is actually better for real-world use?

Standard metrics (like IID accuracy) tell us what a model gets right, but they don't tell us why or how the model perceives the underlying language. Two models might have the same accuracy but arrive at their answers using completely different logic. One might be robust to typos, while the other collapses if a single letter is swapped. To truly compare models—especially when one is a "black box" API—we need to look at their shared invariances: the specific perturbations to an input that leave the model’s judgment unchanged.

Methods: The SCoPE Framework

The authors propose a framework called SCoPE (SHared-Capabilities-thrOugh-Perturbed-Examples). The process works in three steps:

  1. Define a Capability: Choose a linguistic trait to test, such as Synonym-Invariance (replacing words with synonyms) or Typo-Invariance (swapping internal characters in a word).
  2. Generate Invariant Perturbations: Using a "Reference Model" (e.g., BERT), the framework uses a greedy search to find specific changes to a sentence that do not change the Reference Model's output.
  3. Measure the Target: Take those same perturbed sentences and feed them to a "Target Model."

The authors introduce two specific metrics to quantify this relationship:

  • Hard-SCoPE: A binary measure. Does the target model's top prediction (the argmax) stay the same as its original prediction when given the perturbation?
  • Soft-SCoPE: A more granular measure. It looks at the change in the entire probability distribution. Even if the top prediction stays the same, does the "confidence" shift in the same direction and magnitude as the reference model?
Visualizing Soft-SCoPE vs. Hard-SCoPE

Datasets

The study primarily utilizes standard benchmarks to validate its approach:

  • SST2 (Stanford Sentiment Treebank): A standard binary sentiment classification dataset.
  • AG’s News: A multi-class topic classification dataset used to validate results across different tasks.
  • WordNet: Used as the engine for generating synonym-based perturbations.
  • TextAttack: A library used to implement the search and transformation constraints.

Key Insights and Ideas

  • The Asymmetry of Scale: One of the most striking findings is that larger models are "behavioral supersets" of smaller ones. Large models share almost all the invariances found in smaller models, but smaller models do not share the unique invariances of the giants. This suggests that "emergent properties" in LLMs might actually be the accumulation of a vast, diverse library of linguistic invariances.
  • Distillation is a Leaky Bucket: When we "distill" a large model (like BERT) into a smaller one (like DistilBERT), we don't just lose a bit of accuracy. We lose specific robustness. The authors found that DistilBERT loses significantly more Typo-Invariance than Synonym-Invariance, likely because the pre-training objective (Masked Language Modeling) reinforces word-level meaning more than character-level robustness.
  • RLHF Retains More "Soul": When comparing OpenAI’s models, text-davinci-003 (trained with Reinforcement Learning from Human Feedback) shared more invariances with the base GPT-2 than its supervised-only counterpart (text-davinci-002). This suggests RLHF might be better at preserving the underlying behavioral "logic" of language than standard supervised finetuning.

Key Results

  • Accuracy vs. Invariance: BERT and DistilBERT have a small accuracy gap (~3.5%), but their Typo-Invariance alignment is much lower (84%) compared to their Synonym-Invariance (96%). This highlights how standard accuracy masks deeper behavioral differences.
Synonym vs. Typo Invariance between BERT and DistilBERT
  • Scale Trends: In the BERT family, as model size increases (from Tiny to Base), the models become more similar to each other in terms of shared invariances, regardless of which one is used as the reference.
  • Black-Box Insights: Using GPT-2 as a reference, the authors showed that larger InstructGPT models (like davinci) are significantly more "agreeable" to GPT-2's invariances than smaller versions like ada, providing a way to "fingerprint" the behavior of closed-source models.

Limitations and Future Work

  • Search Efficiency: The current method uses a greedy search to find perturbations. This is computationally expensive and requires many queries to the model, which can be costly when testing commercial APIs.
  • Semantic Uncertainty: The paper notes that estimating the output probabilities of instruction-tuned models (which often output text rather than log-probs) is an open challenge. They used a "multiple sample" approach to estimate probabilities, but more robust methods are needed.
  • Expanding Capabilities: The study focused on synonyms, typos, and fairness. Future work could expand this to more complex linguistic features like negation, passive voice, or cross-lingual invariances.

Summary updated

Abstract

An explosion of work in language is leading to ever-increasing numbers of available natural language processing models, with little understanding of how new models compare to better-understood models. One major reason for this difficulty is saturating benchmark datasets, which may not reflect well differences in model performance in the wild. In this work, we propose a novel framework for comparing two natural language processing models by revealing their shared invariance to interpretable input perturbations that are designed to target a specific linguistic capability (e.g., Synonym-Invariance, Typo-Invariance). Via experiments on models from within the same and across different architecture families, this framework offers a number of insights about how changes in models (e.g. distillation, increase in size, amount of pre-training) affect multiple well-defined linguistic capabilities. Furthermore, we also demonstrate how our framework can enable evaluation of the invariances shared between models that are available as commercial black-box APIs (e.g., InstructGPT family) and models that are relatively better understood (e.g., GPT-2). Across several experiments, we observe that large language models share many of the invariances encoded by models of various sizes, whereas the invariances encoded by large language models are only shared by other large models. Possessing a wide variety of invariances may be a key reason for the recent successes of large language models, and our framework can shed light on the types of invariances that are retained by or emerge in new models.
BibTeX
@misc{rawal2023perturbed,
  author = {Ruchit Rawal and Mariya Toneva},
  title = {{Perturbed examples reveal invariances shared by language models}},
  year = {2023},
  eprint = {2311.04166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.04166},
}
Figure