A Memory Augmented Architecture for Continuous Speaker Identification in Meetings

Nikolaos Flemotomos, Dimitrios Dimitriadis

Introduction

Continuous Speaker Identification (CSI) is a critical component in multi-party meeting analysis, tasked with determining the active speaker at any given timestamp based on pre-enrolled voice profiles. Traditional CSI systems typically rely on a pipeline that extracts speaker embeddings (e.g., x-vectors) and compares them to profiles using fixed heuristic distance metrics like Cosine Similarity or Probabilistic Linear Discriminant Analysis (PLDA).

However, these traditional approaches suffer from significant drawbacks. First, fixed metrics often assume specific data distributions that do not account for the variability of real-world meeting environments. Second, they treat audio segments as independent events, ignoring the inherent temporal continuity of human speech. Finally, standard embeddings are often optimized for large-scale speaker verification rather than the fine-grained discrimination required among a small, specific set of meeting participants.

Methods: Relational Memory Core

To address these limitations, this research proposes replacing heuristic metrics with a Relational Memory Core (RMC), a type of Memory-Augmented Neural Network (MANN). By framing speaker identification as a relational reasoning task, the model learns a data-driven distance metric tailored to the specific participants in a meeting.

The architecture processes a sequence consisting of the current audio segment ($x_i$) followed by the speaker profiles ($s_1, \dots, s_N$). Through self-attention mechanisms, the RMC allows the audio segment's representation to interact with the profile memories, effectively learning the "distance" between them in a high-dimensional space.

The proposed RMC-based architecture for speaker identification.

Beyond relational reasoning, the authors incorporate temporal context by expanding the input to include neighboring segments ($x_{i-1}$ and $x_{i+1}$). This allows the model to leverage conversational flow, acting as a learned smoothing filter. Post-processing via median filtering further refines the output by removing erratic, short-term classification errors.

Key Results and Analysis

The RMC-based approach was evaluated on the AMI Corpus and a dataset of internal Microsoft business meetings. To ensure generalization to unseen speakers, the model was pre-trained on the large-scale VoxCeleb dataset with extensive data augmentation (noise and reverberation).

The results demonstrate a substantial improvement over the standard cosine-distance baseline. On the AMI dataset, the RMC achieved a 39.28% relative reduction in Speaker Error Rate (SER). The gains were even more pronounced on real-world internal meetings, where the SER was reduced by 51.84%.

Speaker Error Rate (SER) across different datasets and filter lengths.

As shown in the performance analysis, the inclusion of temporal context ("RMC+context") consistently yields the lowest error rates across various median filter lengths. On the internal meeting set, classification accuracy improved from 70.66% (baseline) to 79.97% with the full RMC+context model. The system also proved robust to varying numbers of participants, ranging from 4 to 15 speakers.

Conclusion and Future Work

By moving away from static distance formulas toward a memory-augmented relational architecture, this work significantly advances the state-of-the-art in continuous speaker identification. The RMC effectively captures both the relational nuances between speaker profiles and the temporal dynamics of natural conversation.

Future research directions include moving toward true diarization by implementing "on-the-fly" profile capturing, which would eliminate the need for pre-enrolled fingerprints. Additionally, exploring more sophisticated transition models could further enhance the system's ability to handle complex speaker overlaps and rapid turn-taking.

Summary updated

Abstract

We introduce and analyze a novel approach to the problem of speaker identification in multi-party recorded meetings. Given a speech segment and a set of available candidate profiles, a data-driven approach is proposed learning the distance relations between them, aiming at identifying the correct speaker label corresponding to that segment. A recurrent, memory-based architecture is employed, since this class of neural networks has been shown to yield improved performance in problems requiring relational reasoning. The proposed encoding of distance relations is shown to outperform traditional distance metrics, such as the cosine distance. Additional improvements are reported when the temporal continuity of the audio signals and the speaker changes is modeled in. In this paper, the proposed method is evaluated in two different tasks, i.e. scripted and real-world business meeting scenarios, where a relative reduction in speaker error rate of 39.28% and 51.84%, respectively, is reported when compared with the baseline.
BibTeX
@inproceedings{flemotomos2020memory,
  author = {Nikolaos Flemotomos and Dimitrios Dimitriadis},
  title = {{A Memory Augmented Architecture for Continuous Speaker Identification in Meetings}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053152},
}
Figure