Introduction
Continuous Speaker Identification (CSI) is a critical component in multi-party meeting analysis, tasked with determining the active speaker at any given timestamp based on pre-enrolled voice profiles. Traditional CSI systems typically rely on a pipeline that extracts speaker embeddings (e.g., x-vectors) and compares them to profiles using fixed heuristic distance metrics like Cosine Similarity or Probabilistic Linear Discriminant Analysis (PLDA).
However, these traditional approaches suffer from significant drawbacks. First, fixed metrics often assume specific data distributions that do not account for the variability of real-world meeting environments. Second, they treat audio segments as independent events, ignoring the inherent temporal continuity of human speech. Finally, standard embeddings are often optimized for large-scale speaker verification rather than the fine-grained discrimination required among a small, specific set of meeting participants.
Methods: Relational Memory Core
To address these limitations, this research proposes replacing heuristic metrics with a Relational Memory Core (RMC), a type of Memory-Augmented Neural Network (MANN). By framing speaker identification as a relational reasoning task, the model learns a data-driven distance metric tailored to the specific participants in a meeting.
The architecture processes a sequence consisting of the current audio segment ($x_i$) followed by the speaker profiles ($s_1, \dots, s_N$). Through self-attention mechanisms, the RMC allows the audio segment's representation to interact with the profile memories, effectively learning the "distance" between them in a high-dimensional space.
Beyond relational reasoning, the authors incorporate temporal context by expanding the input to include neighboring segments ($x_{i-1}$ and $x_{i+1}$). This allows the model to leverage conversational flow, acting as a learned smoothing filter. Post-processing via median filtering further refines the output by removing erratic, short-term classification errors.
Key Results and Analysis
The RMC-based approach was evaluated on the AMI Corpus and a dataset of internal Microsoft business meetings. To ensure generalization to unseen speakers, the model was pre-trained on the large-scale VoxCeleb dataset with extensive data augmentation (noise and reverberation).
The results demonstrate a substantial improvement over the standard cosine-distance baseline. On the AMI dataset, the RMC achieved a 39.28% relative reduction in Speaker Error Rate (SER). The gains were even more pronounced on real-world internal meetings, where the SER was reduced by 51.84%.
As shown in the performance analysis, the inclusion of temporal context ("RMC+context") consistently yields the lowest error rates across various median filter lengths. On the internal meeting set, classification accuracy improved from 70.66% (baseline) to 79.97% with the full RMC+context model. The system also proved robust to varying numbers of participants, ranging from 4 to 15 speakers.
Conclusion and Future Work
By moving away from static distance formulas toward a memory-augmented relational architecture, this work significantly advances the state-of-the-art in continuous speaker identification. The RMC effectively captures both the relational nuances between speaker profiles and the temporal dynamics of natural conversation.
Future research directions include moving toward true diarization by implementing "on-the-fly" profile capturing, which would eliminate the need for pre-enrolled fingerprints. Additionally, exploring more sophisticated transition models could further enhance the system's ability to handle complex speaker overlaps and rapid turn-taking.
Summary updated