Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback

Yisha Wu, Cen Zhao, Yuanpei Cao, Xiaoqing Xu, Yashar Mehdad, Mindy Ji, Claire Na ChengEMNLP 2025

TL;DR

This paper introduces an incremental summarization system for customer support agents that generates concise notes in real-time, reducing context-switching and improving efficiency. By combining a fine-tuned LLM for note generation with a classifier for relevance filtering and an agent feedback loop for continuous improvement, the system achieved a 3% reduction in case handling time and high agent satisfaction.

What Problem Does This Paper Address?

Customer support agents handle complex, often lengthy, multi-source conversations across various channels (phone, chat, email). A significant portion of their time (around 10%) is spent writing summary notes, which are crucial for handoffs, compliance, and audits. This note-taking process exacerbates context-switching and increases overall working time. Existing summarization tools often operate post-conversation or use basic chunking, failing to address the real-time needs of agents and the challenge of accurately capturing critical information from long, unstructured dialogues. The demand for highly accurate summaries is also paramount, as inaccuracies can lead to poor decision-making and customer dissatisfaction.

Methods

The core of the system is a Progressive Note-Taking with Quality Control workflow. This involves:

  • LLM-based Summarization with Continuous Bullet Generation: A fine-tuned Mixtral-8x7B model generates summary bullets incrementally. It uses an iterative prompting strategy that includes task instructions, contextual data, full interaction history, and previously generated bullets. This ensures that only new, incremental information is summarized, and previous edits are preserved.
  • Bullet Relevance Classifier for Filtering: A DeBERTa-based classifier is used to filter out trivial or redundant bullets, retaining only high-value utterances categorized into specific types (e.g., customer issue, agent solution). This classifier achieved strong performance (F1 of 0.895 in production).
  • Agent Edits Learning Framework: This framework enables continuous improvement through two paths:
  • Online Interaction Updates: Agents can directly edit the generated notes in real-time. These edits immediately inform subsequent LLM prompts, ensuring the model uses the most accurate, agent-verified information.
  • Offline Model Refinement: Agent edits are logged and used as training signals. A pipeline involving initial quality assessment, conditional high-quality rewriting, and pairwise preference data generation is used to create a training corpus for further LLM refinement via supervised fine-tuning and preference alignment.

Datasets

The paper primarily uses an internal dataset of production case conversations for training and evaluation. They also mention using the CCNet dataset for pre-training the DeBERTa classifier. For benchmarking, they refer to SAMSum (Gliwa et al., 2019) and DialSum (Chen et al., 2021) datasets.

Key Insights and Ideas

  • Incremental summarization is key for agent efficiency: The most significant insight is that generating notes during the conversation, rather than at the end, dramatically reduces context-switching and saves agent time.
  • Agent feedback is crucial for real-world alignment: The "Agent Edits Learning Framework" is vital. It's not just about collecting edits, but intelligently processing them to refine the model, ensuring it learns to produce summaries that agents actually find useful and accurate.
  • A two-stage filtering approach is effective: Combining an LLM for initial generation with a dedicated classifier for relevance filtering strikes a good balance between capturing information and maintaining conciseness.
  • LLM-as-a-judge can be a powerful evaluation tool: The paper demonstrates that GPT-4o as an LLM judge can outperform human annotators in certain aspects and offers significant throughput advantages for evaluation.

Key Results

  • Reduced Handling Time: The system achieved a 3% reduction in average handling time across all cases, with up to 9% reduction for complex cases (involving 3+ agents or long solving times).
  • High Agent Satisfaction: Surveys showed over 95% satisfaction among agents in English, French, and Spanish.
  • Improved Summary Quality: Offline evaluations showed clear incremental gains in completeness, truthfulness, and overall scores as the model progressed through fine-tuning stages (Mixtral-Base -> Mixtral-NF -> Mixtral-FB). The final Mixtral-FB model achieved an overall score of 0.885.
  • Efficiency Gains: The system reduced the agent's manual note-writing time ratio from 10% to 3%.
  • No Negative Impact on Customer Satisfaction: Customer NPS scores remained neutral, indicating no adverse effect on customer experience.

Limitations and Future Work

  • LLM-Judge Bias: Reliance on GPT-4o as an LLM judge might introduce evaluation biases or inaccuracies compared to human judgments.
  • Bullet Classifier Errors: The classifier can occasionally misclassify content, potentially leading to omissions or irrelevant updates. Continuous retraining is needed.
  • Multilingual Performance: Summarization performance varies across languages, and adapting the system to new languages remains a challenge.
  • Future Work: The authors plan to extend the system to more languages, incorporate multimodal contexts (like images), and further experiment with continuous retraining of the classifier.

Summary updated

BibTeX
@inproceedings{wu2025incremental,
  author = {Yisha Wu and Cen Zhao and Yuanpei Cao and Xiaoqing Xu and Yashar Mehdad and Mindy Ji and Claire Na Cheng},
  title = {{Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure