Zero-Shot Data Maps. Efficient Dataset Cartography Without Model Training

Angelo Basile, Marc Franco-Salvador, Paolo RossoEMNLP 2023 · DOI

Abstract

As NLP datasets scale, label noise and annotation artifacts become significant hurdles. Traditional Dataset Cartography identifies these issues by tracking model training dynamics, but this process is computationally expensive. We present Zero-Shot Data Maps, a method that generates diagnostic maps up to 14x faster by leveraging the sensitivity of zero-shot bi-encoders to label phrasing. Our approach matches or exceeds traditional methods in error detection and data selection while requiring no model training.

Introduction

Modern NLP relies on massive datasets often curated via crowdsourcing or scraping. These "black boxes" frequently contain mislabeled instances that degrade model performance. Dataset Cartography has emerged as a powerful diagnostic tool, plotting instances based on Confidence (mean probability of the true label) and Variability (fluctuation of confidence across training). However, calculating these metrics typically requires training a high-capacity model over multiple epochs, creating a significant bottleneck for large-scale data auditing.

Methodology

Zero-Shot Data Maps replace training epochs with an ensemble of zero-shot bi-encoder models (specifically sentence-t5-large). The core insight is "turning a bug into a feature": zero-shot models are notoriously sensitive to prompt phrasing. By using an ensemble of different but semantically equivalent label descriptions (e.g., "positive," "great," "awesome" for sentiment), the authors simulate the variability typically observed during the learning process.

  • Confidence: Calculated as the average probability assigned to the true label across all different phrasings.
  • Variability: Defined as the standard deviation of those probabilities.
  • Efficiency: Using bi-encoders allows for single-pass document embedding, making the approach feasible for massive datasets compared to cross-encoders.
Figure 6: Zero-Shot Data Maps across different benchmarks, illustrating the distribution of data points based on confidence and variability.

Key Results and Insights

The proposed method achieves a 14x speedup on GPU and an even more dramatic improvement on CPU (28 minutes vs. over 9 hours). Beyond speed, the method offers several advantages:

  1. Error Detection Superiority: On the IMDB error detection task, Zero-Shot maps achieved an Average Precision of 34.7, outperforming the traditional Training Dynamics baseline of 30.2. This suggests zero-shot models may be less prone to "memorizing" noisy labels during diagnosis.
  2. Comparable Data Selection: When selecting a 33% subset of data to train a new model, zero-shot maps performed nearly identically to the much slower traditional maps.
  3. Time-Unrolled Equivalence: The authors argue that an ensemble of zero-shot models prompted with different descriptions is functionally equivalent to a single model at different stages of training.

Limitations

While highly effective for sentiment and topic classification, the method struggles with linguistically complex tasks like CoLA (grammaticality). As shown in the performance distributions, the model lacks the nuance required to assess complex linguistic properties in a zero-shot setting compared to a fully trained model.

Figure 4: F1 score distributions across datasets. The lower performance on CoLA highlights the limitations of zero-shot bi-encoders for complex linguistic tasks.

Conclusion

Zero-Shot Data Maps provide a rapid, training-free alternative for dataset auditing. By exploiting the inherent brittleness of zero-shot models, researchers can identify "hard-to-learn" or mislabeled examples at a fraction of the traditional computational cost, paving the way for cleaner, more reliable large-scale NLP datasets.

Summary updated

BibTeX
@inproceedings{basile2023zero,
  author = {Angelo Basile and Marc Franco-Salvador and Paolo Rosso},
  title = {{Zero-Shot Data Maps. Efficient Dataset Cartography Without Model Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.554},
}
Figure