Introduction
In the current landscape of Large Language Models (LLMs), the mantra "more data is better" is increasingly being challenged. NLP researchers have identified that massive datasets often contain significant noise, redundant information, and "data artifacts" that can hinder model generalization. Identifying which specific data points are truly informative remains a critical challenge. Traditional approaches often rely on single metrics—such as model confidence or loss—to rank data. However, these individual signals are frequently incomplete and fail to capture the full complexity of a dataset's "DNA."
This paper introduces infoVerse, a universal framework designed to characterize datasets by mapping them into a multidimensional space of model-driven meta-information. By leveraging a diverse set of metrics and a sophisticated subset selection method, infoVerse provides a more holistic view of data quality and informativeness.
The infoVerse Framework
The infoVerse pipeline consists of two primary stages: the construction of a multidimensional feature space and the selection of a maximally informative subset.
First, each data point is represented by 23 distinct model-driven metrics. These metrics are categorized into four groups:
- Static Measures: Snapshot-based metrics from a single model state, such as Entropy, Confidence, and Gradient Norms.
- Training Dynamics: Temporal signals tracking how a model's prediction for a sample changes over time, including Forgetting Number and Area Under Margin.
- Model Uncertainty: Disagreement-based metrics derived from multiple models or dropout masks, such as BALD and Variance Ratio.
- Pre-trained Knowledge: Task-agnostic signals derived from pre-trained models like BERT or RoBERTa, including Sentence Density and Pseudo-log-likelihood.
Second, to select the most valuable data, the framework employs Determinantal Point Processes (DPP). Unlike simple ranking methods that might select a cluster of similar "hard" samples, DPP optimizes for both quality (informativeness based on the 23 metrics) and diversity (ensuring the selected points are spread out across the infoVerse space).
The Value of Multidimensionality
A core premise of infoVerse is that different meta-information metrics capture complementary aspects of data. For instance, a sample might be difficult because it is out-of-distribution (low density) or because it is mislabeled (high forgetting). Relying on a single metric would conflate these distinct types of "difficulty."
As shown in the correlation analysis, many of these 23 metrics are weakly correlated or even uncorrelated across different categories. This lack of redundancy justifies the multidimensional approach; by combining these disparate signals, infoVerse can distinguish between truly difficult linguistic cases and simple noise or artifacts.
Empirical Evaluation and Applications
The authors evaluated infoVerse across three major data-centric tasks using a variety of benchmarks (QNLI, SST-5, WinoGrande, AGNEWS, etc.):
- Data Pruning: infoVerse-DPP consistently identifies the most redundant samples. On the WinoGrande dataset, it achieved 74.6% accuracy, significantly outperforming both random sampling (73.1%) and single-metric "hard" selection (69.6%). The framework automatically adapts its strategy based on the pruning ratio, keeping high-variability samples when data is scarce.
- Active Learning: In iterative learning scenarios, infoVerse achieved the lowest average rank (1.3) compared to established baselines like BADGE and ALPS across multiple datasets.
- Human Data Annotation: The framework effectively identifies "linguistic gray areas." Samples flagged by infoVerse as informative showed the highest levels of disagreement among human annotators, suggesting a strong alignment between model-driven difficulty and human cognitive load.
Conclusion
infoVerse represents a significant step toward a more nuanced, data-centric approach to machine learning. By moving beyond single-metric rankings to a multidimensional characterization of data, it enables more efficient training and better dataset curation. While the computational cost of generating 23 metrics is a consideration, the performance gains in pruning and active learning suggest that the investment in characterization pays off in model quality and reduced annotation costs.
Summary updated