infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information

Jaehyung Kim, Yekyung Kim, Karin de Langis, Jinwoo Shin, Dongyeop Kang

Introduction

In the current landscape of Natural Language Processing (NLP), the prevailing wisdom often suggests that "bigger is better" regarding dataset size. However, this focus on volume frequently overlooks a critical reality: not all data points contribute equally to model learning. Large-scale datasets are often saturated with redundant samples, label noise, and "annotation artifacts"—spurious correlations that allow models to achieve high performance without truly mastering the underlying task.

While various methods exist to characterize data quality—such as measuring model confidence or tracking "forgetting events" during training—these approaches are typically one-dimensional. A single metric often fails to distinguish between a complex, high-quality sample and a simple mislabeled one. To address this, we introduce infoVerse, a universal framework that maps the "hidden geography" of datasets using a multidimensional meta-information space.

The infoVerse Framework

The core innovation of infoVerse is the construction of a 23-dimensional feature space derived from four distinct categories of model-driven meta-information:

  1. Static Measures: Metrics captured from a single model snapshot, such as entropy and prediction confidence.
  2. Training Dynamics: Observations of how a model’s relationship with a specific data point evolves over the training process (e.g., variability and the frequency of forgetting events).
  3. Model Uncertainty: Measures of disagreement between different model architectures or different versions of the same model (e.g., Monte-Carlo Dropout).
  4. Pre-trained Knowledge: Insights extracted from models like BERT or RoBERTa before they are fine-tuned on the target task.

By combining these diverse signals, infoVerse reveals clusters and characteristics that remain invisible in traditional semantic spaces. It allows for a nuanced distinction between "hard-but-valuable" samples and "easy-to-mistake" noisy samples.

Strategic Sampling via infoVerse-DPP

To effectively navigate this 23D space, we propose infoVerse-DPP, a sampling method based on the Determinantal Point Process (DPP). Traditional sampling often struggles to balance informativeness with diversity; a model might identify 100 "highly informative" samples that are virtually identical, leading to redundancy. infoVerse-DPP mathematically optimizes for a subset of data points that are both high-quality (informative) and diverse (spread across the infoVerse space), ensuring a more representative and efficient training set.

Experimental Results and Applications

The infoVerse framework was rigorously evaluated across three primary real-world applications using standard NLP benchmarks including NLI (QNLI, RTE), Sentiment Analysis (SST-2, SST-5), and Reasoning (WinoGrande).

1. Data Pruning

In data pruning tasks, the goal is to remove less useful data without sacrificing performance. On the WinoGrande dataset, infoVerse-DPP achieved an average accuracy of 74.6% across various pruning ratios. This significantly outperformed random sampling (73.1%) and single-metric "Hard" sampling (69.6%). Notably, infoVerse-DPP automatically adapts to the pruning ratio: at low ratios, it removes redundant high-confidence samples, while at high ratios, it preserves them to maintain a foundational knowledge base.

2. Active Learning

Active learning involves iteratively selecting the most beneficial samples for labeling. Across SST-2, RTE, and AGNEWS, infoVerse-DPP achieved the lowest average rank (1.3) compared to all baseline methods. By leveraging the complementary effects of training dynamics and model uncertainty, the framework consistently identifies the most informative samples for the next stage of training.

3. Data Annotation

When selecting samples for human annotation, infoVerse-DPP demonstrated superior performance, particularly on the IMP (insult detection) dataset. The samples flagged by the framework as "informative" showed a strong correlation with instances where human annotators disagreed, suggesting that infoVerse successfully identifies subjective or ambiguous cases that require human intervention.

Limitations and Future Directions

While infoVerse offers a powerful tool for dataset characterization, several challenges remain:

  • Model Bias: As a model-driven framework, infoVerse may inherit or amplify biases present in the underlying models used to generate meta-information.
  • Computational Overhead: Generating 23 dimensions of meta-information requires multiple training runs and forward passes. Although this cost is relatively small compared to total training time for large models, it is a consideration for resource-constrained environments.
  • Theoretical Constraints: While DPP sampling is effective in practice, further research is needed to provide theoretical guarantees for optimal subset selection across all possible tasks.

Conclusion

infoVerse moves beyond one-dimensional data assessment by providing a comprehensive, multidimensional map of dataset characteristics. By integrating diverse meta-information and employing diversity-aware sampling, infoVerse enables more efficient data pruning, more effective active learning, and higher-quality data annotation, paving the way for more robust and data-efficient NLP systems.

Summary updated

Abstract

The success of NLP systems often relies on the availability of large, high-quality datasets. However, not all samples in these datasets are equally valuable for learning, as some may be redundant or noisy. Several methods for characterizing datasets based on model-driven meta-information (e.g., model's confidence) have been developed, but the relationship and complementary effects of these methods have received less attention. In this paper, we introduce infoVerse, a universal framework for dataset characterization, which provides a new feature space that effectively captures multidimensional characteristics of datasets by incorporating various model-driven meta-information. infoVerse reveals distinctive regions of the dataset that are not apparent in the original semantic space, hence guiding users (or models) in identifying which samples to focus on for exploration, assessment, or annotation. Additionally, we propose a novel sampling method on infoVerse to select a set of data points that maximizes informativeness. In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines in all applications. Our code and demo are publicly available.
BibTeX
@inproceedings{kim2023infoverse,
  author = {Jaehyung Kim and Yekyung Kim and Karin de Langis and Jinwoo Shin and Dongyeop Kang},
  title = {{infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=3SkCC1vIat},
}
Figure