Introduction
In the evolving landscape of autonomous driving, Referring Multi-Object Tracking (RMOT) has emerged as a critical capability, allowing vehicles to track specific objects based on external instructions. Traditionally, these instructions have been text-based. However, text input is often impractical and unsafe for a driver. Auditory commands offer a more natural and safer interface, yet they introduce significant technical hurdles. Audio signals are inherently unstructured, containing variations in pitch, timbre, and background noise that are difficult to map to dynamic visual pixels.
To address this, we introduce EchoTrack, a framework designed for the novel task of Auditory Referring Multi-Object Tracking (AR-MOT). EchoTrack bridges the modality gap between sound and sight, enabling robust tracking through spoken commands.
The EchoTrack Framework
EchoTrack is an end-to-end architecture that leverages two primary innovations to handle the complexity of auditory signals:
- Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM): Unlike standard models that fuse modalities in the spatiotemporal domain, EchoTrack utilizes Fast Fourier Transforms (FFT) to analyze features in the frequency domain. This allows the model to capture the unique "signature" of auditory cues and adaptively filter visual features more effectively than time-domain methods.
- Audio-visual Contrastive Tracking Learning (ACTL): To ensure semantic alignment, ACTL employs a contrastive learning regime. It pulls matching audio and visual trajectory features closer in a shared embedding space while pushing non-matching pairs apart, ensuring the tracker remains locked on the correct target.
The system utilizes a ResNet50 backbone for visual extraction, a frozen HuBERT-Base model for audio encoding, and a Deformable DETR-style transformer for final tracking output.
Benchmarking AR-MOT
Due to the lack of existing datasets for this task, we developed three new large-scale benchmarks by augmenting existing driving datasets with professional audio annotations:
- Echo-KITTI: 18 videos with 820 audio expressions.
- Echo-KITTI+: An expansion featuring longer, more complex linguistic descriptions to test model limits.
- Echo-BDD: Based on the BDD100K dataset, this challenging benchmark includes 50 videos across diverse weather conditions (rainy, night, overexposure) and complex urban scenes.
Performance Analysis
EchoTrack establishes a new state-of-the-art (SOTA) for auditory tracking. On the Echo-KITTI benchmark, it achieved a Higher Order Tracking Accuracy (HOTA) of 37.14, significantly outperforming the previous best RMOT model, TransRMOT (33.58).
As shown in the performance breakdown, EchoTrack maintains a consistent lead across all environmental conditions. Notably, it demonstrates high resilience in visually challenging scenarios like night and underexposed scenes, where auditory cues provide critical context that text-based baselines lack. Despite its sophisticated frequency-domain processing, the model maintains an inference speed of approximately 10 FPS, making it viable for near-real-time applications.
Conclusion and Future Work
EchoTrack demonstrates that auditory cues are not merely "spoken text" but rich physical signals that can enhance tracking robustness in autonomous driving. While the current model excels in diverse environments, future work will focus on improving performance during long-term occlusions and increasing inference frame rates. Additionally, integrating Large Language Models (LLMs) could further enhance the system's ability to interpret abstract or ambiguous auditory commands.
Summary updated