Auto311: A Confidence-guided Automated System for Non-emergency Call

Zirong Chen, Xutong Sun, Yuanhe Li, Meiyi MaarXiv 2023 · DOI · arXiv

Introduction

Emergency Communication Centers (DECs) are currently facing unprecedented operational pressure. While 911 services are reserved for life-threatening emergencies, 311 systems handle a high volume of municipal non-emergency issues, such as road damage or stolen property. Often, the same personnel manage both streams, meaning a surge in 311 calls can directly delay responses to critical 911 emergencies.

Existing automated dialogue systems, such as AWS Lex or Google DialogFlow, typically rely on rigid, pre-defined flowcharts that struggle with the fluid nature of real-world calls. Research indicates that 72% of callers provide "unintentional additional information" early in a conversation, and incident types shift or multiply in 38% of cases. Auto311 is designed to address these complexities using a multi-modular, confidence-guided architecture that dynamically extracts information and predicts incident types from messy, real-world dialogues.

System Architecture

Auto311 employs a modular design to ensure both efficiency and safety. The system begins with a conversational interface utilizing OpenAI’s Whisper for speech-to-text and Suno-AI Bark for text-to-speech.

The Auto311 system architecture illustrates the flow from the caller through the interface and handover control to the core processing modules.

As shown in the architecture above, the system includes an "always-on" handover control layer. This safety mechanism uses Latent Dirichlet Allocation (LDA) and NLP rules to detect high-urgency keywords or caller frustration, immediately routing the call to a human operator if necessary. The core logic resides in the interaction between Information Itemization and Incident Type Prediction, governed by confidence thresholds ($λ_1, λ_2$).

Confidence-Guided Report Generation

The primary innovation of Auto311 is its use of confidence scores to navigate the dialogue. Instead of a simple classifier, the Incident Type Prediction module uses a multi-layer hierarchical structure with Monte Carlo Dropout to measure consistency across trials.

A detailed walkthrough of the report generation process, showing how information is itemized and incident types are predicted and updated based on confidence scores.

If the system identifies a field or incident type with low confidence ($conf \leq λ$), it proactively asks for clarification. Conversely, if confidence is high, the system strategically skips irrelevant questions to streamline the call. This approach allows the system to "harvest" information from long-winded explanations, filling out future parts of the report before the specific question is even asked. As illustrated in the figure above, the system can dynamically update, merge, or even delete incident types (e.g., shifting from "Damaged Property" to "Lost Stolen") as more context is gathered.

Evaluation and Key Results

The researchers evaluated Auto311 using a unique dataset of 11,796 authentic non-emergency call recordings from the Nashville Department of Emergency Communications. For information itemization, the model was benchmarked against standard datasets like SQuAD and CUAD.

Key performance metrics include:

  • Accuracy: Achieved an average F1-score of 92.54% for incident prediction across categories like noise violations and abandoned vehicles.
  • Consistency: Reached a 0.93 average consistency score in information extraction compared to ground-truth human reports.
  • Efficiency: Emulations demonstrated that the system successfully decreased the number of conversation turns as the length of the caller's initial utterance increased, proving its ability to process proactive information.
  • Superiority over General LLMs: While GPT-3.5 and GPT-4 are powerful, Auto311—fine-tuned on domain-specific data—outperformed them in information itemization by up to 47%.

Conclusion and Future Work

Auto311 demonstrates the potential for task-specific, confidence-guided systems to alleviate the burden on emergency dispatchers. However, several challenges remain. Future work will focus on integrating GIS for better landmark-to-address translation and developing redundancy checks to merge multiple reports regarding the same incident. Currently, the system has been tested via emulations; live, interactive trials with real callers represent the next critical step in validating its real-world utility.

Summary updated

Abstract

Emergency and non-emergency response systems are essential services provided by local governments and critical to protecting lives, the environment, and property. The effective handling of (non-)emergency calls is critical for public safety and well-being. By reducing the burden through non-emergency callers, residents in critical need of assistance through 911 will receive a fast and effective response. Collaborating with the Department of Emergency Communications (DEC) in Nashville, we analyzed 11,796 non-emergency call recordings and developed Auto311, the first automated system to handle 311 non-emergency calls, which (1) effectively and dynamically predicts ongoing non-emergency incident types to generate tailored case reports during the call; (2) itemizes essential information from dialogue contexts to complete the generated reports; and (3) strategically structures system-caller dialogues with optimized confidence. We used real-world data to evaluate the system's effectiveness and deployability. The experimental results indicate that the system effectively predicts incident type with an average F-1 score of 92.54%. Moreover, the system successfully itemizes critical information from relevant contexts to complete reports, evincing a 0.93 average consistency score compared to the ground truth. Additionally, emulations demonstrate that the system effectively decreases conversation turns as the utterance size gets more extensive and categorizes the ongoing call with 94.49% mean accuracy.
BibTeX
@misc{chen2023auto,
  author = {Zirong Chen and Xutong Sun and Yuanhe Li and Meiyi Ma},
  title = {{Auto311: A Confidence-guided Automated System for Non-emergency Call}},
  year = {2023},
  eprint = {2312.14185},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.14185},
}
Figure