Abstract
Abhinav Gupta, Devendra Singh, Greig A Cowan, N Kadhiresan, Siddharth Srivastava, Yagneswaran Sriraja, Yoages Kumar Mantri. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).
2025.
Introduction
In the highly regulated banking sector, deploying Large Language Models (LLMs) for customer-advisor conversation summarization presents significant governance and safety challenges. While LLMs excel at condensing information, they are prone to "hallucinations," speaker attribution errors, and potential privacy leaks. In a financial context, an incorrect summary could lead to regulatory violations or misinformed financial commitments. The AUTOSUMM framework addresses these challenges by bridging the gap between experimental LLM performance and the rigorous standards required for enterprise-grade deployment.
The AUTOSUMM Framework
The AUTOSUMM architecture is a production-ready system composed of four specialized modules designed to ensure reliability, quality, and safety.
- Core Summarization Module: This module leverages OpenAI’s GPT-3.5 Turbo, utilizing Chain of Thought (CoT) prompting to maintain logical consistency. It includes a semantic-chunking algorithm to handle long conversations and a fine-tuned BERT model as an extractive fallback.
- Data Management Module: It monitors transcript quality and employs semantic clustering (SBERT + UMAP + HDBSCAN) to track thematic shifts and detect data drift.
- Risk Management (DreamCatcher): A dedicated module for hallucination detection using multi-layered verification.
- Model Performance Management: This module tracks structural metrics and advisor edits, providing a continuous feedback loop for model improvement.
DreamCatcher: Hallucination Detection
A cornerstone of the AUTOSUMM framework is the "DreamCatcher" module, which serves as a multi-layered "hallucination hunter." Recognizing that no single metric can capture all factual inconsistencies, the authors implemented a three-tier verification process to ensure the summary is grounded in the transcript.
- Syntactic Layer: Performs named entity recognition (NER) to compare dates, monetary values, and names between the transcript and the summary.
- Semantic Layer: Utilizes Sentence-BERT embeddings to verify that summary sentences align semantically with the original transcript segments.
- Entailment Layer: Employs a BART-based Natural Language Inference (NLI) model to ensure the summary logically follows from the source text.
Evaluation and Results
The framework was validated using proprietary data from NatWest Group, including a reference dataset of 600,000 sentences and production data of approximately 12,000 conversations per month. Human evaluation involved over 150 summaries reviewed by 22 advisors.
Key results include:
- Factual Consistency: A 94% consistency rate achieved through the DreamCatcher module.
- Operational Efficiency: An average time saving of 15 minutes per call for advisors, removing the burden of manual note-taking.
- Production Accuracy: 89% of live summaries required no human edits, with only 1% requiring major corrections.
- User Satisfaction: 85% of advisors rated the summaries with "Strongly Agree" for quality and coherence.
Conclusion
AUTOSUMM demonstrates that LLMs can be safely deployed in high-stakes environments when supported by robust risk management and human-in-the-loop feedback. By treating advisor edits as a primary data source and implementing multi-layered hallucination checks, the framework ensures both efficiency and regulatory compliance. Future work will focus on scaling this framework across diverse banking departments and enhancing real-time anomaly detection to further reduce the need for human oversight.
Summary updated