Low-Confidence Gold: Refining Low-Confidence Samples for Efficient Instruction Tuning

Hongyi Cai, Jie Li, Mohammad Mahdinur Rahman, Wenzhen DongEMNLP 2025

TL;DR

This paper introduces "Low-Confidence Gold" (LCG), a novel framework for efficiently tuning Large Language Models (LLMs) by intelligently filtering instruction-response datasets. LCG identifies high-quality, diverse data points using a semi-supervised approach, leading to improved LLM performance with significantly less data and computational cost.

What Problem Does This Paper Address?

Instruction fine-tuning (IFT) is crucial for making LLMs follow instructions effectively. However, existing IFT datasets often suffer from low quality and misleading content, which can hinder model performance. While some methods try to address this by using other LLMs to score data or employing clustering, they can lead to biased datasets or fail to consistently balance data quality and diversity. This paper tackles the challenge of creating high-quality, diverse instruction datasets efficiently, reducing the need for extensive manual annotation and computational resources, ultimately improving LLM performance.

Methods

The LCG framework operates in two main steps:

  1. Clustering and Pseudo-labeling:
    • The Alpaca_52k dataset is first clustered into $K$ semantic groups using K-means. This step aims to capture the inherent diversity of instruction patterns.
  2. A "coreset" of instruction-response pairs is selected based on proximity to cluster centroids. These selected pairs are used to generate pseudo-labels, serving as initial high-confidence training data.
  3. Low-Confidence Gold Selection:
    • A lightweight classifier (either Multinomial Naive Bayes or DistilBERT) is trained on this core-set for a very short period (3 epochs). This "early-stopped" training prevents overfitting and encourages the classifier to be uncertain about more complex or nuanced examples.
  4. The classifier then predicts confidence scores for the remaining data. Samples with low confidence scores are identified as "Low-Confidence Gold" – these are considered challenging, high-quality examples that are most beneficial for LLM training. These low-confidence samples are then selected for the final filtered dataset.

Datasets

  • Training and Evaluation: The primary dataset used is Alpaca_52k.
  • Cross-Dataset Validation: The WizardLM dataset was also used for cross-dataset validation to demonstrate the generalizability of the LCG approach.

Key Insights and Ideas

  • Low Confidence = High Value: The core idea is that samples where a simple classifier is least confident are precisely the ones that are most valuable for training LLMs. These are the "hard" examples that push the model's understanding.
  • Semi-Supervised Efficiency: By using a small, early-stopped classifier trained on pseudo-labels derived from clustering, the method avoids the need for extensive human annotation or large, proprietary LLMs for scoring.
  • Balancing Quality and Diversity: The initial clustering step ensures that diverse instruction patterns are represented, while the low-confidence selection refines this to focus on high-quality examples within each cluster.
  • Underfitting as a Feature: Deliberately underfitting the classifier is key. It ensures the model doesn't become too specialized on the initial core-set, making its low-confidence predictions more indicative of genuinely challenging data.

Key Results

  • Superior Performance: Fine-tuning LLMs (Mistral-7b-v0.3 and LLaMa3-8b) on LCG-filtered subsets (e.g., 6K samples) achieved superior performance compared to existing methods.
  • MT-Bench Gains: LCG with MultinomialNB achieved the highest MT-bench score (5.086) for Mistral-7b, a 14.5% improvement over the previous best. LCG with DistilBERT also showed significant gains on LLaMA3-8b.
  • Efficiency: The method demonstrated strong performance even with very small subsets (e.g., 1K examples), highlighting its efficiency in data curation.
  • Robustness: LCG showed consistent improvements across various benchmarks (Hellaswag, MMLU, GSM8k, ARC) and maintained effectiveness on the larger WizardLM dataset, indicating good cross-dataset generalization.
  • Synergy with RLHF: Combining LCG with Reinforcement Learning from Human Feedback (RLHF) further boosted performance, especially on reasoning tasks.

Limitations and Future Work

  • Initial Data Selection Effort: While the classifier training is efficient, the initial step of selecting data with annotated pseudo-labels still requires some effort.
  • Dataset Bias: The method might still be influenced by the original biases and task distributions present in the initial dataset, potentially limiting its effectiveness if the source data is fundamentally flawed.
  • Future Work: The paper suggests that LCG can serve as an effective initial filter for more advanced refinement techniques like RLHF, indicating potential for multi-stage filtering pipelines.

Summary updated

BibTeX
@inproceedings{cai2025low,
  author = {Hongyi Cai and Jie Li and Mohammad Mahdinur Rahman and Wenzhen Dong},
  title = {{Low-Confidence Gold: Refining Low-Confidence Samples for Efficient Instruction Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure