From Noise to Clarity: Filtering Real and LLM-Generated Samples for Enhanced Intent Detection

Junbao Huang, Weizhen Li, Peijie Huang, Yuhong XuEMNLP 2025

TL;DR

This paper introduces SFTM, an interpretable framework that filters both real and LLM-generated dialogue samples for intent detection. By evaluating samples on "Distribution Diversity" and "Sample Authenticity," SFTM significantly improves model performance, especially in low-resource settings, by prioritizing quality over quantity.

What Problem Does This Paper Address?

Intent detection, a crucial component of dialogue systems, often suffers from data quality issues. Acquiring sufficient, accurately labeled, and representative real-world data is expensive and time-consuming. While Large Language Models (LLMs) offer a promising avenue for data augmentation, the synthetic data they generate can be noisy, mislabeled, or lack diversity due to the LLMs' inherent limitations. This noise can degrade the performance of intent detection models. Existing LLM-based data curation methods often act as black boxes, lacking interpretability and controllability. This paper addresses the challenge of effectively filtering both real and LLM-generated dialogue samples to improve the quality and robustness of intent detection models.

Methods

The core of the proposed solution is the Sample Filter by Topic Modeling (SFTM) framework. SFTM is an interpretable dual-criteria mechanism that evaluates samples based on two key dimensions:

  • Distribution Diversity: This module assesses how well a sample's latent representation aligns with a prior distribution. It uses Kullback-Leibler (KL) divergence to measure the difference between the encoder's posterior distribution and a standard Gaussian prior. This encourages diverse and regularized latent representations, preventing overfitting and mode collapse. Reconstruction loss (using Mean Squared Error) is also employed to ensure the latent representation captures essential features.
  • Sample Authenticity: This module leverages the availability of labels to assess how well a sample aligns with its assigned intent. It uses a supervised classification task, employing a linear layer to classify reconstructed samples and analyzing the resulting logits. High logits suggest easily classifiable samples with limited training value, while low logits might indicate label errors.

SFTM then uses a Sample Filtering Strategy that incorporates "seed samples" (a small set of high-quality real samples) to dynamically adapt thresholds for both diversity and authenticity. This allows for a nuanced evaluation and selection of high-quality samples from both real and synthetic sources.

Datasets

The paper uses two datasets for evaluation:

  • Banking-77: A fine-grained, utterance-level dataset in the banking domain with 77 intents.
  • CMCC-34: A dialogue-level dataset for Chinese customer service interactions, known for containing significant noise due to speech transcription.

The authors also generate synthetic samples using GPT-4o-mini for data augmentation.

Key Insights and Ideas

  • Quality over Quantity: The research strongly suggests that for intent detection, the quality of training data is more critical than sheer quantity. SFTM achieves superior performance with significantly fewer samples than methods that rely on large volumes of synthetic data.
  • Interpretability Matters: Unlike black-box LLM-as-a-Judge approaches, SFTM's dual-criteria evaluation (Distribution Diversity and Sample Authenticity) provides interpretable insights into why samples are filtered, enabling better control over data curation.
  • LLMs Can Hallucinate Intent: Even with few-shot prompting, LLMs can generate semantically inconsistent samples that misrepresent the intended label, highlighting the necessity of post-generation verification.
  • Noise is a Major Bottleneck for LLMs: The paper demonstrates that LLMs can exhibit counterintuitive behavior (e.g., underperforming older models) on noisy datasets. High-quality filtered samples are crucial for mitigating these noise-induced biases and unlocking LLMs' full potential.

Key Results

  • Full-shot Experiments: SFTM achieved significant performance improvements, with maximum gains of 1.5% on Banking-77 and 3.59% on CMCC-34, outperforming baselines like DA-GPT and LRSL.
  • Less is More: SFTM achieved better results with substantially fewer samples (e.g., 680 synthetic samples for CMCC-34 vs. 3,400 for DA-GPT).
  • Few-shot Experiments: SFTM consistently outperformed state-of-the-art methods like ICDA and DA-Judge in few-shot settings, demonstrating its effectiveness in low-resource scenarios. For instance, on Banking-77, SFTM achieved 86.29% elimination of low-quality synthetic samples.
  • Ablation Studies: Filtering real samples improved performance by up to 1%, while filtering synthetic samples yielded gains of up to 2.09%. Combining filtered real and synthetic samples provided the largest boost.

Limitations and Future Work

  • Focus on Data Augmentation: The current work is primarily focused on data augmentation for intent detection. The authors suggest that exploring this paradigm for other NLU/NLG tasks is a promising future direction.
  • Alternative Evaluation Methods: While SFTM uses specific metrics, the paper acknowledges that other methods for evaluating sample quality could be investigated.
  • Domain Specificity: The current framework is applied to intent detection. Further research could explore its applicability to other downstream tasks and domains.

Summary updated

BibTeX
@inproceedings{huang2025from,
  author = {Junbao Huang and Weizhen Li and Peijie Huang and Yuhong Xu},
  title = {{From Noise to Clarity: Filtering Real and LLM-Generated Samples for Enhanced Intent Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure