Auto311: A Confidence-Guided Automated System for Non-emergency Calls

Zirong Chen, Xutong Sun, Yuanhe Li, Meiyi Ma

TL;DR

This paper introduces Auto311, the first automated system designed to handle non-emergency (311) calls. By dynamically predicting incident types and extracting key information with confidence guidance, Auto311 aims to streamline emergency response and ensure critical 911 calls receive faster attention.

What Problem Does This Paper Address?

Local governments are increasingly pressured to manage essential services like emergency response with fewer resources. Non-emergency calls, typically handled by 311 systems, consume significant dispatcher time. This burden can delay responses to critical 911 calls. Furthermore, non-emergency calls often involve complex scenarios: callers may provide extraneous details, and incidents can shift in nature or involve multiple types simultaneously. Existing automated systems often rely on rigid dialogue flows, making them ill-suited for the dynamic and varied nature of these calls. Auto311 aims to bridge this gap by providing an intelligent, adaptive system for non-emergency call handling.

Methods

Auto311 is built around two core components: incident type prediction and information itemization, all guided by a novel confidence measurement approach.

  • Confidence Measurement: Instead of relying on traditional model output probabilities, Auto311 measures confidence through internal consistency across multiple model runs using Monte Carlo Dropout. This provides a more robust measure of certainty, especially for complex textual outputs.
  • Incident Type Prediction: This module uses a hierarchical structure to identify multiple incident types within a single call, leveraging the full conversational context.
  • Information Itemization: This component extracts relevant details from caller utterances to populate case reports. It focuses on the latest utterance and uses a custom text comparison metric that prioritizes semantic similarity and keyword overlap over traditional metrics like BLEU.
  • Confidence-Guided Report Generation and Dialogue Optimization: This overarching module uses the confidence scores from the other components to dynamically update case reports and strategically guide the conversation. If confidence is low, it prompts for clarification; if high, it moves on. This ensures reports are filled with high-confidence information and optimizes follow-up questions.
  • Handover Control: A crucial safety feature, this module continuously monitors for signs of urgency or caller distress, rerouting calls to human operators when necessary.

Datasets

The primary dataset used in this research consists of 11,796 real-world non-emergency (311) call recordings collected in Nashville, Tennessee, in collaboration with the Department of Emergency Communications (DEC). These audio recordings were transcribed using speaker diarization. The paper also references standard NLP benchmarks like SQuAD and CUAD for evaluating information itemization components.

Key Insights and Ideas

  • The Value of Extraneous Information: Callers frequently provide more details than explicitly asked for. Auto311's ability to leverage this "unintentional additional information" is key to its efficiency.
  • Dynamic Incident Types: Non-emergency calls are not static. Incidents can evolve, and callers might mention multiple issues (e.g., a damaged car that is also stolen). Auto311's architecture is designed to handle this fluidity.
  • Confidence as a Dialogue Driver: The paper introduces a novel way to measure confidence based on internal consistency, which is then used to intelligently guide the conversation, reducing unnecessary turns and ensuring accuracy.
  • Beyond Traditional Text Metrics: Standard text comparison metrics like BLEU are insufficient for the nuanced task of extracting specific, yet concise, information for case reports. A custom metric focusing on keyword and semantic overlap proved more effective.

Key Results

  • Incident Type Prediction: Auto311 achieved an average F-1 score of 92.54% for incident type prediction, outperforming other models, including BERT, when confidence guidance was applied.
  • Information Itemization: The system demonstrated an average consistency score of 0.93 for completed reports, significantly outperforming LLMs like GPT-3.5 and GPT-4 on both archived and recent datasets.
  • System-Level Performance: Emulations showed that Auto311 effectively decreases conversation turns and categorizes ongoing calls with an overall accuracy of 94.49%.
  • Specific Incident Types: For binary incident types like "Illegal Parking," Auto311 achieved 100% F-1 scores, highlighting its precision with well-defined categories.

Limitations and Future Work

  • Address Validation: The current system extracts location information but does not perform explicit address validation, which is critical for emergency response. Future work could integrate geocoding or landmark-based validation.
  • Redundant Calls: The system currently treats all incoming calls equally, even if they report the same incident. Future work could explore strategies to identify and leverage novel information from repeated reports without dismissing them entirely.
  • Deployment: While the system shows strong performance in simulations, real-world deployment and continuous learning from evolving city knowledge are the next steps.
  • Confidence Measurement Refinement: While the proposed confidence metric is effective, further research into its application across diverse domains and its interplay with different machine learning models is suggested.

Summary updated

Abstract

Emergency and non-emergency response systems are essential services provided by local governments and critical to protecting lives, the environment, and property. The effective handling of (non-)emergency calls is critical for public safety and well-being. By reducing the burden through non-emergency callers, residents in critical need of assistance through 911 will receive a fast and effective response. Collaborating with the Department of Emergency Communications (DEC) in Nashville, we analyzed 11,796 non-emergency call recordings and developed Auto311, the first automated system to handle 311 non-emergency calls, which (1) effectively and dynamically predicts ongoing non-emergency incident types to generate tailored case reports during the call; (2) itemizes essential information from dialogue contexts to complete the generated reports; and (3) strategically structures system-caller dialogues with optimized confidence. We used real-world data to evaluate the system's effectiveness and deployability. The experimental results indicate that the system effectively predicts incident type with an average F-1 score of 92.54%. Moreover, the system successfully itemizes critical information from relevant contexts to complete reports, evincing a 0.93 average consistency score compared to the ground truth. Additionally, emulations demonstrate that the system effectively decreases conversation turns as the utterance size gets more extensive and categorizes the ongoing call with 94.49% mean accuracy.
BibTeX
@inproceedings{chen2024auto,
  author = {Zirong Chen and Xutong Sun and Yuanhe Li and Meiyi Ma},
  title = {{Auto311: A Confidence-Guided Automated System for Non-emergency Calls}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30199},
}
Figure