Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings

Taous Iatariene, Alexandre Guérin, Romain SerizelarXiv 2025 · DOI · arXiv

Abstract

Real-time speaker tracking in multi-speaker environments is often hindered by the latency of traditional speaker embedding extractors, which require several seconds of audio to reliably identify voices. This paper presents a low-latency method for tracking multiple speakers by extracting speaker embeddings from audio segments as short as 250ms. By combining Knowledge Distillation (KD) with spatial beamforming, the authors train a "student" model that significantly outperforms standard models on brief, overlapping speech segments. Experimental results on the LibriJump dataset demonstrate that the student model achieves an Association Accuracy (AssA) of 54.8% on 250ms contexts, paving the way for identity-aware, real-time speaker tracking.

Introduction

In dynamic multi-speaker environments like teleconferences, tracking both the location and identity of participants is a significant challenge. Speakers frequently move while silent or pause during conversation, breaking the "spatial continuity" that traditional tracking systems rely on. To maintain identity across these gaps, systems utilize speaker embeddings—neural network-generated voice fingerprints. However, standard embedding extractors require long audio windows to be effective, introducing high latency and increasing the risk of signal degradation from overlapping speech. This work addresses these issues by developing a pipeline capable of extracting highly accurate embeddings from very short temporal contexts.

Methodology

The proposed approach utilizes a teacher-student framework and spatial filtering to optimize performance for low-latency applications.

Knowledge Distillation for Short Contexts

The authors employ a pre-trained ECAPA-TDNN "teacher" model, which produces high-quality embeddings from long, clean audio signals. A "student" model is then trained to mimic the teacher's latent space while being restricted to short, random audio crops (ranging from 250ms to 8s). This distillation process forces the student model to learn robust identity features from minimal temporal information.

Overview of the proposed teacher-student training pipeline and multichannel beamforming integration.

Beamforming and Blockwise Processing

To handle overlapping speech, the system integrates spatial predictions from an underlying tracker to apply Delay-and-Sum beamforming. This isolates the target speaker's audio before embedding extraction. Furthermore, the system processes audio in fixed, short temporal blocks (e.g., 256ms) for identity reassignment, rather than waiting for full speech fragments to conclude, thereby minimizing system latency.

Experimental Results

The system was evaluated using the LibriJump dataset, specifically designed to test tracking systems with intermittent speakers and discontinuous spatial tracks.

Short-Context Superiority

The student model demonstrated a clear advantage in short-context scenarios. On a 250ms window, the student model achieved an AssA of 54.8%, significantly outperforming the teacher model's 46.8% and the baseline tracker's 38.1%.

Quantitative comparison of Teacher and Student model performance across various temporal contexts.

Model Efficiency and Limitations

Efficiency tests showed that the student model is highly compressible; halving the channel size from 1024 to 512 resulted in only a minor performance drop (max 1.7%). However, the system remains sensitive to simultaneous speech. In 2-speaker scenes, performance dropped to 49.4% AssA, highlighting the difficulty of isolating voices during heavy cross-talk. While the low-latency "blockwise" method currently trails the higher-latency "fragment" method in complex scenes, it represents a significant step toward real-time viability.

Conclusion

This paper demonstrates that Knowledge Distillation can successfully adapt speaker embedding extractors for low-latency tracking. By training a student model to focus on identity features within 250ms windows, the proposed system achieves superior performance over traditional long-context models in short-duration evaluations. Future research will focus on improving robustness against overlapping speakers and developing even more compact network architectures for edge deployment.

Summary updated

BibTeX
@misc{iatariene2025towards,
  author = {Taous Iatariene and Alexandre Guérin and Romain Serizel},
  title = {{Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings}},
  year = {2025},
  eprint = {2508.14115},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.14115},
}
Figure