Introduction
In demonstration-based learning, Pretrained Language Models (PLMs) are provided with a few input-output examples to guide their performance on a new query. While it is intuitive to assume models learn task-specific knowledge from these examples, recent findings show that even "meaningless" demonstrations—such as those with random labels or tokens—can significantly improve performance. This paper investigates this paradox through the lens of Structural Causal Models (SCMs), proposing that random demonstrations act as interventions that break harmful biases.
Causal Framework and Methodology
The authors treat demonstration-based learning as a causal inference problem. They map the relationships between demonstration tokens ($D$), context ($C$), entities ($E$), and the final prediction ($Y$) using SCMs.
Identifying the Backdoor Path
Standard, human-crafted demonstrations often create a "backdoor path" ($G \to D \to X$), where unobserved confounders ($G$) introduce bias. For instance, if a specific word frequently co-occurs with a label in a small sample, the model may learn a superficial pattern rather than the underlying task logic.
Intervention via Randomness
Random demonstrations are interpreted as a causal intervention on the variable $D$. By shuffling or replacing tokens, the influence of the confounder is severed, forcing the model to ignore misleading correlations. The authors propose four types of interventions:
- Random: Uniformly sampled tokens from the entire vocabulary.
- Rand-S: Tokens sampled from the unique words in the small support set.
- Rand-W: Weighted sampling favoring entity-related words from the support set.
- Rand-E: Replacing standard demonstrations with random sequences of actual entities and coherent tokens from the support set.
Experimental Results
The researchers evaluated these methods on sequence labeling benchmarks including CoNLL03 (NER), OntoNotes 5.0, and CoNLL00 (chunking). They also used specialized control datasets, NRB (Name Regularity Bias) and WTS, to diagnose whether models rely on entity names versus local context.
Key Findings
- Superiority of Rand-E: The "Rand-E" method consistently outperformed standard demonstrations. On CoNLL03, Rand-E achieved an F1 score of 47.32, compared to 45.86 for standard examples and 28.71 for no demonstrations.
- Breaking Name Bias: Standard demonstrations often trick models into over-relying on specific entity names. Randomizing the context around these entities (as seen in the WTS and NRB results) helps the model focus on surrounding linguistic cues to determine labels.
- Support Set Importance: The most effective results came from sampling tokens from the task-relevant support set rather than the general vocabulary, suggesting that the distribution of relevant tokens is more critical than their specific semantic arrangement.
Discussion and Limitations
This study demonstrates that "meaningless" demonstrations are not useless; they provide valid distributional information without the harmful noise of specific, misleading sentence structures. However, the scope is currently limited to sequence labeling tasks and encoder-only models like BERT and RoBERTa. Future work is needed to determine if these causal interventions translate to generative tasks in multi-billion parameter Large Language Models (LLMs). Additionally, these findings apply specifically to few-shot learning, as zero-shot settings lack a support set for sampling.
Summary updated