Utilizing Multimodal Data for Edge Case Robust Call-sign Recognition and Understanding

Alexander Blatt, Dietrich KlakowarXiv 2024 · DOI · arXiv

Abstract

In Air Traffic Control (ATC), misidentifying a call-sign can lead to critical safety incidents. Standard AI assistants often fail during "edge cases" characterized by high noise, audio clipping, or complete signal loss. This paper presents Call-sign-Command Recovery (CCR), a multimodal system that integrates a novel text model, CallSBERT, with real-time aircraft surveillance data. By fusing linguistic analysis with spatial priors—mapping specific commands to aircraft coordinates—CCR achieves robust performance even in the absence of clear voice transcripts.

Introduction

The reliability of Call-sign Recognition and Understanding (CRU) is paramount in ATC. However, real-world communications are frequently degraded by high Word Error Rates (WER), clipping of initial call-signs, and intermittent signal loss. Existing state-of-the-art models typically rely on clean transcripts, making them fragile in operational environments. This research addresses these vulnerabilities by proposing a multimodal architecture that uses the physical context of the airspace to supplement verbal communication.

Methodology: The CCR Architecture

The proposed CCR system employs a dual-branch approach to identify call-signs, combining linguistic processing with spatial awareness.

1. The Text Branch: CallSBERT

CallSBERT is a lightweight model based on Sentence-BERT (SBERT). Unlike traditional models that attempt to generate call-sign strings directly, CallSBERT utilizes contrastive learning to rank a list of known surveillance call-signs against the received transcript. This approach is significantly more efficient, requiring 63% fewer parameters and training four times faster than previous encoder-decoder architectures.

2. The Command Branch and Multimodal Fusion

The "heavy lifter" of the system is the multimodal integration. A command classifier identifies the instruction type (e.g., "turn left"), which is then processed by the Command Distribution Module (CDM). The CDM uses 3D spatial data—latitude, longitude, and altitude—to calculate the probability of each aircraft in the vicinity being the intended recipient of that specific command.

The CCR architecture integrates a text-based CallSBERT branch with a spatial Command Distribution Module to recover call-signs from noisy or incomplete transcripts.

Spatial Priors as a Safety Net

A key innovation of this work is the use of spatial priors. Because certain ATC instructions are highly correlated with specific locations in the airspace (e.g., landing clearances near runways), the aircraft's position serves as a powerful secondary identifier.

Spatial command distributions visualize how specific instructions correlate with aircraft positions in the airspace, providing a robust secondary signal for identification.

The system fuses the scores from the text and command branches, allowing it to "recover" the correct call-sign even when the transcript is garbled or missing. The authors found that training on "trash" data—intentionally noisy and clipped transcripts—further enhanced the model's real-world applicability without sacrificing performance on clean data.

Key Results and Robustness

The CCR architecture demonstrates significant improvements in operational robustness across several metrics:

  • Edge Case Performance: The system provided up to a 15% increase in accuracy for edge case scenarios compared to text-only baselines.
  • Resilience to Clipping: While text-only models fail when the initial call-sign is clipped, the command branch maintains high identification accuracy by leveraging the instruction type.
  • Zero-Transcript Recovery: In extreme scenarios with 100% transcript loss, the CCR model achieved nearly 40% accuracy by relying solely on the command type and aircraft coordinates.

Conclusion

By combining efficient contrastive learning with spatial context, the CCR system offers a robust solution for ATC communication assistance. This multimodal approach effectively mitigates the risks associated with signal degradation, providing a reliable safety net for controllers. Future work will explore expanding this methodology to other safety-critical domains with strict spatial-verbal correlations, such as maritime navigation or military operations.

Summary updated

Abstract

Operational machine-learning based assistant systems must be robust in a wide range of scenarios. This hold especially true for the air-traffic control (ATC) domain. The robustness of an architecture is particularly evident in edge cases, such as high word error rate (WER) transcripts resulting from noisy ATC recordings or partial transcripts due to clipped recordings. To increase the edge-case robustness of call-sign recognition and understanding (CRU), a core tasks in ATC speech processing, we propose the multimodal call-sign-command recovery model (CCR). The CCR architecture leads to an increase in the edge case performance of up to 15%. We demonstrate this on our second proposed architecture, CallSBERT. A CRU model that has less parameters, can be fine-tuned noticeably faster and is more robust during fine-tuning than the state of the art for CRU. Furthermore, we demonstrate that optimizing for edge cases leads to a significantly higher accuracy across a wide operational range.
BibTeX
@misc{blatt2024utilizing,
  author = {Alexander Blatt and Dietrich Klakow},
  title = {{Utilizing Multimodal Data for Edge Case Robust Call-sign Recognition and Understanding}},
  year = {2024},
  eprint = {2412.20467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20467},
}
Figure