Abstract
Fine-tuning Large Language Models (LLMs) effectively requires a balanced mixture of high-quality data across various capabilities, such as reasoning, mathematics, and coding. However, real-world datasets are frequently "domain-undetermined," lacking the clean, normalized labels necessary for traditional data-mixing strategies. We present Diversity as a Reward (DAAR), a self-supervised framework that enables LLMs to autonomously curate their training data. By utilizing an LLM’s internal embeddings to predict semantic entropy, DAAR identifies high-value samples that bridge disparate domains. Experimental results demonstrate that DAAR achieves state-of-the-art performance across multiple benchmarks, particularly in STEM fields, while significantly reducing computational overhead compared to GPT-based data selection methods.
1. Introduction
The performance of a fine-tuned LLM is heavily dependent on the composition of its training set. Current methodologies for data preparation generally fall into two categories: data mixing and data selection. Data mixing requires predefined domain proportions, which is often unfeasible for unlabeled, heterogeneous datasets. Conversely, data selection methods typically prioritize "quality"—defined as how closely a sample matches a target distribution—often at the expense of diversity. This can lead to models that are over-specialized or lack the breadth required for multi-domain tasks.
To address these challenges, we introduce DAAR, a framework that treats diversity as a reward signal. DAAR leverages the LLM’s own internal representation of data to select a diverse, high-performing subset from an unlabeled pool, bypassing the need for human-annotated labels or expensive external model scoring.
2. The DAAR Framework
DAAR assigns the LLM a "dual identity," where it serves as both the probe for data selection and the target for fine-tuning. The framework consists of three primary components:
2.1 Model-Aware Centroid Synthesis
Instead of relying on external human labels, DAAR allows the LLM to define its own "domain anchors." Using zero-shot prompting, the model describes specific domains (e.g., "Mathematics," "Coding") and iteratively generates synthetic samples. These samples are used to establish the center of these domains within the model's own embedding space, ensuring the selection process is aligned with the model's internal geometry.
2.2 The Diversity Probe
A lightweight Multi-Layer Perceptron (MLP) is attached to the LLM’s frozen internal layers. This probe is trained to classify samples into the synthesized domains. The core innovation lies in the use of predictive entropy as a selection metric. When the probe exhibits high entropy (i.e., "confusion") regarding a sample's domain, it indicates that the sample is semantically rich and resides in the overlap between multiple domains. These high-entropy samples are prioritized as high-value data points.
2.3 Closed-Loop Selection
The LLM utilizes these entropy scores to filter the data pool, selecting the top 20% of samples. This curated subset is then used for the final fine-tuning process, creating a closed-loop system where the model's internal perception informs its own learning curriculum.
3. Experimental Setup
3.1 Datasets and Models
The seed data for the experiments was constructed from a mixture of Dolly-15k (common sense), Cot-en (reasoning), Math-Instruct (mathematics), and Code-Alpaca (coding). The framework was evaluated using several model families, including Llama 3.1-8B, Qwen2-7B, Qwen2.5-7B, and the reasoning-focused Qwen3-8B.
3.2 Evaluation Benchmarks
Models were tested across seven major benchmarks to assess comprehensive capabilities:
- Knowledge: NQ and TriviaQA
- Reasoning: HellaSwag
- Mathematics: GSM8K and MATH
- Coding: MBPP and HumanEval
4. Results and Discussion
4.1 Performance Gains
DAAR consistently outperformed nine baseline methods, including Alpagasus and DEITA, achieving new state-of-the-art average scores across all tested model families. The most significant improvements were observed in STEM-related tasks. For the Qwen2.5 model, DAAR improved mathematical reasoning by 27% and coding proficiency by 7.4% compared to random data selection.
4.2 Internal Perception vs. Human Labels
A critical finding of this research is that an LLM’s internal understanding of data may be superior to human categorization for the purposes of training. In ablation studies, using the LLM’s own pseudo-labels (generated via clustering) resulted in better fine-tuning outcomes than using ground-truth human labels. This suggests that the model's geometric embedding space captures nuances relevant to its learning process that human labels may overlook.
4.3 Computational Efficiency
DAAR is highly efficient, operating on frozen embeddings with a minimal MLP probe. This architecture results in 70% lower GPU usage and a 2.5x speed increase compared to methods that require API calls to external models like GPT-4 for data scoring.
5. Conclusion
Diversity as a Reward (DAAR) provides a robust, efficient, and self-supervised path for fine-tuning LLMs on unlabeled data. By prioritizing semantic entropy and internal model intuition, DAAR identifies the most beneficial samples for enhancing model capabilities. Future work will explore the application of DAAR to more specialized technical domains and its potential in developing fully autonomous, self-evolving AI systems that curate their own training pipelines in real-time.
Summary updated