Auto311: A Confidence-Guided Automated System for Non-emergency Calls

Zirong Chen, Xutong Sun, Yuanhe Li, Meiyi Ma

Abstract

Emergency communication centers face significant staffing shortages while managing millions of 311 non-emergency calls annually. This paper introduces Auto311, a multi-modular automated system designed to handle complex, unpredictable 311 dialogues. By employing a novel confidence-guided framework based on Monte Carlo Dropout, Auto311 dynamically predicts incident types and itemizes critical information, optimizing conversation flow in real-time. Evaluated on a real-world dataset from Nashville, TN, Auto311 achieves a 92.54% F-1 score in incident detection and outperforms general-purpose LLMs like GPT-4 in domain-specific information extraction.

Introduction

Emergency communication centers (DECs) are under immense pressure. While 911 handles life-threatening emergencies, 311 services manage municipal issues like road damage or stolen property. However, the sheer volume of these calls—nearly 240 million annually in the U.S.—often overwhelms staffing levels that have shrunk significantly since the pandemic.

Existing automated dialogue frameworks (like AWS Lex or Google DialogFlow) struggle with 311 calls because they rely on rigid, pre-defined flowcharts. Real-world callers are unpredictable: they often provide "unintentional additional information" (72% of cases) or report multiple, shifting incident types (38% of cases). Auto311 addresses the need for a flexible, context-aware system that can adapt to these evolving narratives.

Overview of the Auto311 system workflow from caller to first responder.

Methodology

The researchers developed a multi-modular architecture called Auto311, which consists of several key components:

  1. Conversational Interface: Uses OpenAI’s Whisper for robust speech-to-text and Suno-AI’s Bark for realistic text-to-speech.
  2. Incident Type Prediction: Instead of a simple flat classifier, this module uses a multi-layer hierarchical structure to iteratively identify the most likely incident types from the full conversation context.
  3. Information Itemization: This module extracts specific details (like addresses or descriptions) to fill out case reports using extractive question-answering and binary classification.
  4. Confidence-Guided Optimization: Using Monte Carlo Dropout, the system measures the consistency of model outputs. If the system is highly confident, it skips redundant questions; if confidence is low, it strategically asks for clarification.
  5. Always-on Handover Control: A safety mechanism that uses Latent Dirichlet Allocation (LDA) and rule-based detection to identify high-urgency keywords or frustrated callers for immediate transfer to a human operator.
Detailed architecture of the Auto311 confidence-guided framework.

Experimental Setup

The study is grounded in a substantial real-world dataset comprising 11,796 authentic non-emergency call recordings from the Department of Emergency Communications in Nashville, TN. For the itemization task, Auto311 was compared against standard NLP benchmarks (SQuAD, CUAD, TriviaQA) and large language models including GPT-3.5 and GPT-4.

Results and Discussion

Auto311 demonstrates significant improvements over baseline models and general-purpose LLMs:

  • Incident Prediction: Achieved an average F-1 score of 92.54% across 11 incident categories.
  • Information Itemization: Earned a 0.93 average consistency score, surpassing GPT-3.5 (0.63) and GPT-4 (0.65).
  • Dialogue Efficiency: Emulations showed that the system effectively reduces conversation turns by leveraging "over-shared" information to proactively fill report fields.
  • Adaptability: The system successfully handled "incident shifts" within three conversational turns.

The results highlight that domain specificity trumps general LLMs in this context. Auto311 views "over-sharing" as an optimization opportunity rather than a source of confusion, using the extra data to shorten calls while maintaining 94.49% accuracy in ongoing categorizations.

Conclusion

Auto311 represents a significant step forward in automating municipal emergency services. While current evaluations rely on emulated conversations, the system's robust performance in incident detection and information extraction suggests it can substantially reduce the burden on human dispatchers. Future work will focus on full-scale real-world deployment and enhancing the system's ability to adapt to evolving city knowledge.

Summary updated

Abstract

Emergency and non-emergency response systems are essential services provided by local governments and critical to protecting lives, the environment, and property. The effective handling of (non-)emergency calls is critical for public safety and well-being. By reducing the burden through non-emergency callers, residents in critical need of assistance through 911 will receive a fast and effective response. Collaborating with the Department of Emergency Communications (DEC) in Nashville, we analyzed 11,796 non-emergency call recordings and developed Auto311, the first automated system to handle 311 non-emergency calls, which (1) effectively and dynamically predicts ongoing non-emergency incident types to generate tailored case reports during the call; (2) itemizes essential information from dialogue contexts to complete the generated reports; and (3) strategically structures system-caller dialogues with optimized confidence. We used real-world data to evaluate the system's effectiveness and deployability. The experimental results indicate that the system effectively predicts incident type with an average F-1 score of 92.54%. Moreover, the system successfully itemizes critical information from relevant contexts to complete reports, evincing a 0.93 average consistency score compared to the ground truth. Additionally, emulations demonstrate that the system effectively decreases conversation turns as the utterance size gets more extensive and categorizes the ongoing call with 94.49% mean accuracy.
BibTeX
@inproceedings{chen2024auto,
  author = {Zirong Chen and Xutong Sun and Yuanhe Li and Meiyi Ma},
  title = {{Auto311: A Confidence-Guided Automated System for Non-emergency Calls}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30199},
}
Figure