Abstract
Emergency communication centers face significant staffing shortages while managing millions of 311 non-emergency calls annually. This paper introduces Auto311, a multi-modular automated system designed to handle complex, unpredictable 311 dialogues. By employing a novel confidence-guided framework based on Monte Carlo Dropout, Auto311 dynamically predicts incident types and itemizes critical information, optimizing conversation flow in real-time. Evaluated on a real-world dataset from Nashville, TN, Auto311 achieves a 92.54% F-1 score in incident detection and outperforms general-purpose LLMs like GPT-4 in domain-specific information extraction.
Introduction
Emergency communication centers (DECs) are under immense pressure. While 911 handles life-threatening emergencies, 311 services manage municipal issues like road damage or stolen property. However, the sheer volume of these calls—nearly 240 million annually in the U.S.—often overwhelms staffing levels that have shrunk significantly since the pandemic.
Existing automated dialogue frameworks (like AWS Lex or Google DialogFlow) struggle with 311 calls because they rely on rigid, pre-defined flowcharts. Real-world callers are unpredictable: they often provide "unintentional additional information" (72% of cases) or report multiple, shifting incident types (38% of cases). Auto311 addresses the need for a flexible, context-aware system that can adapt to these evolving narratives.
Methodology
The researchers developed a multi-modular architecture called Auto311, which consists of several key components:
- Conversational Interface: Uses OpenAI’s Whisper for robust speech-to-text and Suno-AI’s Bark for realistic text-to-speech.
- Incident Type Prediction: Instead of a simple flat classifier, this module uses a multi-layer hierarchical structure to iteratively identify the most likely incident types from the full conversation context.
- Information Itemization: This module extracts specific details (like addresses or descriptions) to fill out case reports using extractive question-answering and binary classification.
- Confidence-Guided Optimization: Using Monte Carlo Dropout, the system measures the consistency of model outputs. If the system is highly confident, it skips redundant questions; if confidence is low, it strategically asks for clarification.
- Always-on Handover Control: A safety mechanism that uses Latent Dirichlet Allocation (LDA) and rule-based detection to identify high-urgency keywords or frustrated callers for immediate transfer to a human operator.
Experimental Setup
The study is grounded in a substantial real-world dataset comprising 11,796 authentic non-emergency call recordings from the Department of Emergency Communications in Nashville, TN. For the itemization task, Auto311 was compared against standard NLP benchmarks (SQuAD, CUAD, TriviaQA) and large language models including GPT-3.5 and GPT-4.
Results and Discussion
Auto311 demonstrates significant improvements over baseline models and general-purpose LLMs:
- Incident Prediction: Achieved an average F-1 score of 92.54% across 11 incident categories.
- Information Itemization: Earned a 0.93 average consistency score, surpassing GPT-3.5 (0.63) and GPT-4 (0.65).
- Dialogue Efficiency: Emulations showed that the system effectively reduces conversation turns by leveraging "over-shared" information to proactively fill report fields.
- Adaptability: The system successfully handled "incident shifts" within three conversational turns.
The results highlight that domain specificity trumps general LLMs in this context. Auto311 views "over-sharing" as an optimization opportunity rather than a source of confusion, using the extra data to shorten calls while maintaining 94.49% accuracy in ongoing categorizations.
Conclusion
Auto311 represents a significant step forward in automating municipal emergency services. While current evaluations rely on emulated conversations, the system's robust performance in incident detection and information extraction suggests it can substantially reduce the burden on human dispatchers. Future work will focus on full-scale real-world deployment and enhancing the system's ability to adapt to evolving city knowledge.
Summary updated