Tracking of Intermittent and Moving Speakers : Dataset and Metrics

Taous Iatariene, Alexandre Guérin, Romain SerizelarXiv 2025 · DOI · arXiv

Abstract

Sound source tracking in real-world environments faces a significant hurdle: speakers are often intermittent, moving while silent and creating discontinuous spatial tracks. Current tracking systems and evaluation metrics often fail to account for these "jumps," leading to fragmented identities and poor global performance. This paper introduces LibriJump, a synthetic dataset designed to isolate this problem, and adapts the Higher Order Tracking Accuracy (HOTA) framework from computer vision to provide a more nuanced evaluation of audio tracking identity assignment.

Introduction

In typical multi-talker scenarios, such as corporate meetings or social gatherings, participants do not speak continuously. A person might stop talking, move to a different part of the room to consult a whiteboard or change seats, and then resume speaking. For an automated tracking system, this behavior creates a "discontinuous spatial track."

Most existing research in sound source tracking (SST) assumes either continuous acoustic activity or predictable, linear motion during silent intervals. When these assumptions are violated—as in the case of an intermittent moving speaker—trackers often lose the speaker's identity. Upon the speaker's reappearance at a new location, the system frequently assigns a brand-new ID, failing to recognize it as the same individual. This fragmentation is a major barrier for downstream applications like automated diarization and speech enhancement, which rely on consistent identity tracking.

Furthermore, standard audio metrics like the Track Swap Rate (TSR) evaluate errors on a strict frame-by-frame basis. While useful for localizing specific failures, they fail to capture the global severity of identity mix-ups. They cannot distinguish between a system that consistently splits one person's speech into multiple IDs (high fragmentation) and one that mistakenly merges different people together (low precision).

Methodology: Adapting HOTA for Audio

To address the limitations of current metrics, the authors propose a shift from local, frame-level evaluation to global association metrics. They adapt the Association component of the Higher Order Tracking Accuracy (HOTA) framework, which has become a standard for Multi-Object Tracking (MOT) in the computer vision community.

The core of this adaptation involves replacing the visual Intersection-over-Union (IoU) similarity measure—used to match bounding boxes—with an angular distance metric based on the Direction of Arrival (DoA). This allows for the calculation of three key global association metrics tailored for audio:

  • Association Accuracy (AssA): A comprehensive score that balances the system's ability to maintain correct identities across the entire recording.
  • Association Precision (AssPr): A measure of identity purity. It indicates if multiple ground-truth speakers are incorrectly merged into a single predicted track. High precision means a predicted track rarely contains speech from more than one person.
  • Association Recall (AssRe): A measure of identity completeness. It indicates if a single ground-truth speaker is incorrectly split into multiple predicted tracks. High recall means a person's entire speech history is captured under a single ID.

These metrics were benchmarked using two common tracking paradigms: a Bayesian Particle Filter (PF) tracker, which relies on probabilistic spatial updates, and a deep learning-based Neural Network (NN) tracker.

Datasets

The study utilizes two primary datasets to benchmark performance under discontinuous conditions:

  1. LibriJump (New): A novel synthetic dataset provided in First Order Ambisonics (FOA) format. It features 1 to 3 speakers who remain static while active but "jump" to new, random positions during silent intervals. By using LibriSpeech audio and simulated shoebox room impulse responses, the authors created a controlled environment to isolate the impact of spatial jumps on tracking logic.
  2. LOCATA (Existing): A standard dataset of real-world recordings. To test the metrics on real data, the authors modified Task 3 (moving speakers) by artificially zeroing out voice activity periods (creating "Task 3*"). This forces spatial discontinuities in otherwise continuous real-world tracks, simulating the intermittent speaker problem in a complex acoustic environment.

Key Insights and Results

The evaluation revealed that current tracking approaches are highly vulnerable to intermittent activity:

The Failure of Spatial Proximity

Both Bayesian and Neural trackers handle continuous movement effectively but degrade significantly when faced with discontinuous tracks. The spatial jump breaks the core proximity-based assumptions used for identity management. On the modified LOCATA Task 3*, the Particle Filter's Track Swap Rate jumped to 0.28 $s^{-1}$, meaning an ID swap occurred roughly every 3 seconds. Its Association Recall (AssRe) dropped to 47%, indicating that more than half of a single speaker's track was fragmented across different IDs.

Fragmentation vs. Merging Errors

On the single-speaker LibriJump dataset, both trackers showed near-perfect Association Precision (~100%), as there were no other speakers to merge with. However, AssRe was consistently low (around 55%), confirming that trackers redundantly generated new IDs every time a speaker "jumped."

In multi-speaker scenarios (3-speaker LibriJump), the Neural tracker's architectural constraints (it was fixed to output a maximum of 2 IDs) caused its AssPr to plummet to 36.6%. The HOTA metrics effectively captured this failure mode, where the system was forced to merge distinct speakers into single tracks due to its internal limitations.

Hyperparameter Sensitivity

For the Particle Filter tracker, the maximum allowed IDs ($K_{max}$) proved to be a critical hyperparameter. High $K_{max}$ values led to excessive fragmentation; the tracker would spawn a new ID rather than waiting for a previously seen speaker to reappear after a silence. This suggests that current Bayesian trackers lack the "memory" needed to bridge silent gaps.

Conclusion and Future Work

This work demonstrates that frame-level metrics are insufficient for understanding the global behavior of audio trackers in realistic scenarios. By providing the LibriJump dataset and the adapted HOTA metrics, the authors offer a more rigorous framework for evaluating identity persistence.

The results serve as a call to action for the audio community. To maintain identity across silences and jumps, next-generation trackers must move beyond purely spatial cues. Future systems will likely need to integrate non-spatial information, such as voice embeddings or biometric signatures, to bridge the gaps created by intermittent speech and ensure robust tracking in dynamic environments.

Summary updated

BibTeX
@misc{iatariene2025tracking,
  author = {Taous Iatariene and Alexandre Guérin and Romain Serizel},
  title = {{Tracking of Intermittent and Moving Speakers : Dataset and Metrics}},
  year = {2025},
  eprint = {2506.11145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.11145},
}
Figure