How Machine Learning is Improving U.S. Navy Customer Support

Michael Powell, Jamison A. Rotz, Kevin D. O'MalleyAAAI 2020 · DOI

Abstract This paper, originally presented at the Thirty-Second Innovative Applications of Artificial Intelligence Conference (IAAI-20), details the successful real-world deployment of a machine learning pipeline within the U.S. Navy’s support infrastructure. By implementing a custom-built Natural Language Processing (NLP) system, the Navy transformed a manual, labor-intensive ticketing process into a streamlined, automated workflow. The system utilizes Support Vector Machines (SVMs) to categorize and route over 1,000 daily support requests, achieving a 74% reduction in ticket assignment time and a 35% reduction in overall "touch time."

Introduction

The U.S. Navy operates "NAVY 311," a 24/7/365 customer support center that serves as a single point of entry for IT issues, logistics, and shipboard equipment maintenance. Historically, this operation relied on human agents to manually read and route plain-text email requests. With over 1,000 requests arriving daily, this manual process created significant bottlenecks. The challenge was to build an automated system capable of operating within unique constraints: high security (encryption and classification), a specialized military lexicon, and the need for a system that could be updated daily without massive computational overhead.

Methodology

The authors developed a five-step machine learning pipeline inspired by Explicit Semantic Analysis (ESA) to ground the NLP model in domain-specific knowledge:

  1. Archive Generation: A massive corpus was built using Wikipedia articles for general knowledge, alongside internal standard operating procedures (SOPs), FAQs, and previously closed support tickets.
  2. Library Mining: From this archive, a library of approximately 300,000 unique words and phrases was generated.
  3. Vectorization: Incoming requests are converted into numeric representations using TF-IDF (Term Frequency-Inverse Document Frequency), weighting unique domain terms more heavily than common stop words.
  4. Similarity Scoring: The system computes correlation coefficients between new requests and archive documents, transforming sparse vectors into dense "similarity-score" vectors.
  5. Classification: An ensemble of Support Vector Machines (SVMs) processes these scores to predict the correct category and routing destination.

To ensure reliability, the system employs a 95% confidence threshold. If the SVM's prediction confidence falls below this mark, the system declines to auto-populate the field, deferring to a human agent.

Key Results

The pilot study, spanning five months and processing approximately 17,000 tickets, yielded impressive efficiency gains across all primary metrics. By automating the initial triage and routing, the Navy significantly reduced the time required to handle each request.

Table 1: Efficiency gains comparing the automated model against manual processing.

As shown in the table above, the "Time to Assign" saw the most dramatic improvement, dropping by 74%. Overall "Touch Time"—the actual time an agent spends working on a ticket—decreased by 35%, while the total "Time to Close" was reduced by 60%. At the 95% confidence interval, the system achieved an overall accuracy of 94.5%, with routing-specific fields reaching up to 98% accuracy.

Architectural Insights: SVMs vs. Deep Learning

A critical finding of the research was that specialized, simpler architectures can outperform generic deep learning models in niche environments. The team compared their custom SVM ensemble against over 100 different Deep Learning (TensorFlow) configurations.

Figure 1: Accuracy comparison across various fields for different machine learning architectures.

While the deep learning models performed well, the SVMs provided comparable accuracy with significantly higher efficiency. Training the SVMs took only 9 minutes, compared to up to 6 hours for the neural networks. This speed is vital for production environments, as it allows for daily model updates to account for emergent fleet issues. Furthermore, generic cloud-based NLP APIs were found unsuitable due to the Navy’s requirement for air-gapped, highly secure processing environments.

Limitations and Future Directions

The system's primary limitation stems from "noise" in historical training data, such as human agents lazily marking tickets as "General" when more specific categories were available. To address this, the team implemented post-processing data-scrubbing routines to normalize ticket populations.

Future work aims to expand this capability into an interactive correspondence engine (chatbot) to provide real-time solutions to sailors. Additionally, the team plans to use NLP to crawl Security Technical Implementation Guides (STIGs) to proactively identify security risks and predict future resource demand spikes based on historical trends.

Summary updated

BibTeX
@inproceedings{powell2020how,
  author = {Michael Powell and Jamison A. Rotz and Kevin D. O'Malley},
  title = {{How Machine Learning is Improving U.S. Navy Customer Support}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i08.7023},
}
Figure