Ambiguous Examples Help, But Full Data Wins: Revisiting Dataset Cartography for Extractive Adversarial QA

Paul K. MandalarXiv 2026 · arXiv

TL;DR

While the "Dataset Cartography" framework suggests that training on a small subset of "ambiguous" examples can outperform full-dataset training for out-of-distribution (OOD) tasks, this paper finds that the rule doesn't hold for extractive Question Answering (QA). In the stricter setting of span prediction, ambiguous examples are better than random subsets, but they cannot replace the robustness gained from training on the full dataset.

What Problem Does This Paper Address?

Large NLP models often achieve high benchmark scores by exploiting "shortcuts"—like simple word matching—rather than truly understanding the text. Dataset Cartography was introduced as a way to diagnose these issues by mapping training data into three regions based on how a model behaves during training:

  1. Easy-to-learn: High confidence, low variability.
  2. Ambiguous: High variability (the model flips back and forth on these).
  3. Hard-to-learn: Low confidence, low variability.

A provocative claim from the original 2020 cartography paper was that training on just the ambiguous 33% of a dataset could actually result in better OOD performance than training on the full 100%. However, that claim was largely tested on classification tasks (e.g., "Does this sentence contain the answer?").

This paper asks: Does this "ambiguous beats full" effect survive in extractive QA? In extractive QA, the model doesn't just classify; it must identify the exact start and end points of an answer span within a passage, making it a much more complex and "stricter" task.

Methods

The author, Paul K. Mandal, systematically revisited the cartography claim across seven different encoder families (including BERT, RoBERTa, ELECTRA, and DeBERTa).

  1. Mapping: For each model, they computed training dynamics over three epochs on the SQuAD v1.1 dataset to calculate "confidence" (mean probability of the gold span) and "variability" (standard deviation of that probability).
  2. Subsetting: They created 33% subsets based on the rankings (Easy, Ambiguous, Hard) and compared them against a 33% random baseline and the 100% full dataset.
  3. Evaluation: To test robustness, they used Adversarial SQuAD (AddSent and AddOneSent), which inserts distracting sentences into the text that look like they might be the answer but aren't.

Datasets

  • Training: SQuAD v1.1 (the standard "clean" extractive QA dataset).
  • Evaluation: Adversarial SQuAD (AddSent and AddOneSent). These are specifically designed to break models that rely on lexical overlap shortcuts.

Key Insights and Ideas

  • Task Formulation is the Deciding Factor: The "ambiguous beats full" phenomenon seems to be an artifact of simpler classification tasks. When a model has to rank every possible span in a passage, the breadth of the full dataset provides a variety of answer types and contexts that a 33% subset simply cannot match.
  • Ambiguous is still "Better than Random": If you are forced to prune your dataset due to compute constraints, cartography is a great tool. The ambiguous 33% consistently outperformed the random 33% in almost every test. It’s a genuine signal for data selection, just not a "full data killer."
  • Hard Examples are not "Adversarial": Interestingly, training on "hard-to-learn" examples (the ones the model is consistently wrong about) performed worse than random. In extractive QA, these "hard" rows might just be noisy, mislabeled, or simply beyond the model's architectural capacity, rather than being "difficult but informative."

Key Results

  • Ambiguous vs. Random: The ambiguous 33% subset beat the random 33% subset in 14 out of 14 model-evaluation cells for Exact Match (EM) accuracy.
  • Full vs. Ambiguous: The full 100% dataset beat the ambiguous 33% subset in 12 out of 14 cells.
  • The Performance Gap: On average, training on the full dataset provided a gain of 3.40 points in Exact Match and 4.10 points in F1 score over the ambiguous-only training.
  • The Exception: The high-performing DeBERTa-v3-base was the only model where the ambiguous subset nearly tied with the full dataset, suggesting that as models get more powerful, they might become more efficient at learning from smaller, high-signal subsets.

Limitations and Future Work

  • Scope: The study focused on SQuAD v1.1 and didn't include "unanswerable" questions (SQuAD 2.0) or generative QA.
  • Hyperparameters: The author used a fixed fine-tuning protocol across all models. It’s possible that per-model hyperparameter tuning could shift the results slightly.
  • Human Audit: The paper did not perform a human audit of the "hard" examples to confirm if they are truly label errors or just very difficult linguistic puzzles.
  • Future Direction: A major open question is "transferable cartography"—can we use a small, cheap model (like ELECTRA-small) to create a data map that successfully selects training data for a massive, expensive model?

Summary updated

Abstract

Dataset cartography argues that training dynamics can reveal regions of a dataset that matter differently for learning: easy-to-learn examples aid optimization, hard-to-learn examples often expose annotation problems, and ambiguous examples contribute strongly to out-of-distribution generalization. A striking implication of the original work is that a model trained on only 33% ambiguous examples can outperform full-data training on several OOD evaluations, including a QNLI reformulation of SQuAD evaluated on Adversarial SQuAD. We revisit this claim in a stricter setting: open-ended extractive question answering, where models must predict answer spans rather than classify question-sentence pairs. For each encoder family, we compute clean-source SQuAD training dynamics, train on full, random, easy-ranked, ambiguous-ranked, and hard-ranked subsets, and evaluate robustness on adversarial-only AddSent and AddOneSent rows. Ambiguous-ranked 33% subsets beat random 33% subsets on exact-match accuracy in all 14 model-evaluation cells and on F1 in 12/14 cells. However, full-data training beats ambiguous-ranked training in 12/14 exact-match cells and 12/14 F1 cells, with mean full-minus-ambiguous gains of 3.40 EM and 4.10 F1. The most thoroughly replicated ELECTRA-small panel is also the main exception to the ambiguous-over-random F1 gains. These results show that clean-source cartography remains useful for reduced-data selection, but its OOD advantage does not carry over as a general replacement for full-data training in extractive adversarial QA.
BibTeX
@misc{mandal2026ambiguous,
  author = {Paul K. Mandal},
  title = {{Ambiguous Examples Help, But Full Data Wins: Revisiting Dataset Cartography for Extractive Adversarial QA}},
  year = {2026},
  eprint = {2611.19763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19763},
}
Figure