Learning from Failure: Improving Meeting Summarization without Good Samples

Ke Wang, Xiutian Zhao, Wei PengAAAI 2024 · DOI

TL;DR

This paper introduces "Score Tuning," a novel framework for training AI summarization models using only low-quality or "bad" samples, paired with numerical human ratings. This approach bypasses the need for expensive, high-quality "good" samples, making it ideal for real-world industrial settings where such data is scarce. The method demonstrates significant improvements in meeting summarization across multiple languages and shows promise for other NLP tasks.

What Problem Does This Paper Address?

The effectiveness of modern AI language models, particularly for tasks like meeting summarization, heavily relies on large amounts of high-quality, task-specific training data. However, in practical industrial applications, obtaining such data is often challenging due to privacy concerns, resource limitations, and the sheer difficulty of annotating data accurately. For meeting summarization, public datasets are limited, and private meeting transcripts are sensitive. Existing methods like Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) require either "good" samples or pairwise comparisons of "good" vs. "bad" samples, which are difficult to acquire at scale. This paper addresses the critical gap of how to effectively train and improve summarization models when only low-quality or imperfect data is available, and human feedback is limited to simple numerical ratings.

Methods

The core innovation is Score Tuning, a cold-start tuning framework designed to learn from imperfect data. Instead of relying on "gold standard" summaries, Score Tuning utilizes asynchronous, numerical human feedback on generated summaries. The process involves:

  1. Collecting Human Feedback: Users provide numerical scores (1-100) for generated summaries of meeting transcripts. This feedback is asynchronous, meaning each summary is rated independently.
  2. Training with Two Losses: The model is trained by minimizing two key loss functions:
  3. Confidence-Aware Loss ($L_{ca}$): This loss encourages the model to predict human-assigned scores for given transcript-summary pairs. It penalizes the model more heavily when it makes confident but incorrect score predictions, promoting accuracy.
  4. Hindsight-Score Loss ($L_{hs}$): This loss trains the model to generate a summary that corresponds to a given human-assigned score. By minimizing the difference between newly generated summaries and previously generated ones for the same score, the model learns to associate specific scores with summary qualities.
  5. Fixed-Score Inference: After training, the model is prompted to generate a summary for a given transcript, specifically targeting the highest possible score (100), which is assumed to correspond to the best quality summary.

This approach is inspired by Chain of Hindsight (CoH) but differs by using numerical ratings instead of pairwise comparisons and by incorporating the confidence-aware loss.

Datasets

The authors used a combination of datasets for their experiments:

  • In-house Dataset: A newly constructed dataset for meeting summarization, derived from internal company meeting recordings. Notably, this dataset lacks gold-standard summaries, making it ideal for testing the cold-start capabilities of Score Tuning. Transcripts were generated using an ASR system and were not post-edited to mimic real-world conditions.
  • VCSum Dataset: An existing Chinese meeting summarization dataset (Wu et al., 2023) was augmented. This dataset includes query-summary pairs and transcripts.
  • TL;DR Dataset: An English dataset derived from Reddit posts and their summaries (Völske et al., 2017), used to test language independence.
  • CWMT 2018 Dataset: Used for machine translation experiments to explore the transferability of Score Tuning.

Key Insights and Ideas

  • Leveraging "Bad" Data is Viable: The most significant insight is that AI models can be effectively trained and improved using data that is explicitly not high-quality, as long as there's a way to quantify its quality (here, numerical scores). This dramatically expands the potential data sources for model training.
  • Numerical Scores as Rich Feedback: Asynchronous numerical scores provide a more nuanced signal than binary "chosen/rejected" preferences. This allows for finer-grained learning of quality associations.
  • Confidence Matters: The confidence-aware loss is crucial. It helps the model learn to be more certain about its predictions when it's likely correct, and more cautious when it's uncertain, leading to more robust learning.
  • Transferability Beyond Summarization: The successful application of a variation of Score Tuning to machine translation suggests that this framework is not limited to summarization and could be a generalizable technique for aligning LLMs with human preferences across various NLP tasks.

Key Results

  • Improved Summarization Performance: Score Tuning significantly outperformed baseline methods (Pre-trained, SFT, RLHF, CoH) on both the in-house and VCSum datasets, achieving higher ROUGE scores and better human evaluation ratings across accuracy, coherence, coverage, and overall quality. For example, on the VCSum dataset, Score Tuning achieved a +5.30 average ROUGE score improvement over the pre-trained baseline.
  • Higher Win Rates in Human Evaluation: On the in-house dataset, where no gold summaries were available, Score Tuning achieved significantly higher win rates against other methods in human preference tests.
  • Language Independence: The method showed strong performance on an English meeting summarization task, demonstrating its generalizability across languages.
  • Machine Translation Gains: In machine translation, Score Tuning also showed competitive results, outperforming other baseline methods, though the BLEU score improvements were modest, suggesting potential limitations of BLEU as a sole metric for evaluating this type of tuning.
  • Iterative Improvement: The model demonstrated continuous improvement through iterative training cycles, with the most significant gains occurring in the initial rounds, highlighting its effectiveness in cold-start scenarios.

Limitations and Future Work

  • Inter-rater Agreement: While the method demonstrated effectiveness, the authors noted moderate inter-rater agreement (ICC values of 0.57 and 0.61) for the numerical feedback. This suggests that while the approach works, improving the consistency of human ratings could further enhance performance.
  • Controllable and Multi-Aspect Scores: The current framework uses a single overall quality score. Future work could explore introducing controllable and multi-aspect scores (e.g., separate scores for accuracy, conciseness, etc.) to allow for more nuanced control.
  • Broader Transferability: The authors suggest investigating the method's transferability to other tasks where numerical human feedback is applicable.
  • Interchangeability of Feedback Types: Exploring whether absolute numerical feedback can be used to construct valid pairwise comparison data, and vice versa, is another promising research direction.

Summary updated

BibTeX
@inproceedings{wang2024learning,
  author = {Ke Wang and Xiutian Zhao and Wei Peng},
  title = {{Learning from Failure: Improving Meeting Summarization without Good Samples}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29883},
}
Figure