MCE 2018: The 1st Multi-Target Speaker Detection and Identification Challenge Evaluation

Suwon Shon, Najim Dehak, Douglas A. Reynolds, James R. Glass

Introduction

While traditional speaker recognition research focuses on 1-to-1 verification, real-world applications like call center fraud prevention require "multi-target" detection. In these scenarios, a system must monitor incoming calls against a "blacklist" of thousands of known fraudsters. The 1st Multi-target Speaker Detection and Identification Challenge (MCE 2018) was established to benchmark performance in this high-stakes, large-scale environment, moving beyond small cohorts to target sets exceeding 3,000 individuals.

The Multi-Target Detection Framework

The challenge defines two primary tasks: Top-S detection (determining if a caller is on the blacklist) and Top-1 identification (identifying the specific individual). The baseline system provided by the organizers utilizes i-vectors, which are fixed-length representations of speaker characteristics extracted from 13,000 hours of unlabeled telephone speech.

System architecture for multi-target detection and identification.

As shown in the architecture above, the system processes an input utterance through a stack of $S$ detectors. Scores are then normalized using Multi-target Normalization (M-Norm) to account for variability across target speakers. Finally, a ranking and decision stage determines whether to accept a hypothesis based on a threshold $\theta$. Successful participants improved upon this by using Probabilistic Linear Discriminant Analysis (PLDA) backends and Neural Network (NN) fusion.

The MCE18 Dataset

The MCE18 dataset is one of the largest public resources for this task, featuring 26,017 total speakers. The blacklist consists of 3,631 actual fraudulent callers from real-world call center recordings, while the background set includes over 22,000 non-blacklist speakers. Due to privacy constraints, the data was provided as 600-dimensional i-vectors rather than raw audio. This setup forces a focus on backend modeling and normalization rather than front-end feature engineering or linguistic analysis.

Scaling Challenges and Results

A primary finding of MCE 2018 is the "scaling penalty." As the number of target speakers in the blacklist grows, the complexity of the verification task increases, leading to higher error rates. Unlike simple group classification, blacklisted speakers share no common vocal traits, making this a massive-scale verification problem.

Equal Error Rate (EER) performance degradation as the blacklist size increases.

The figure above illustrates that the Equal Error Rate (EER) for both Top-S and Top-1 detectors increases significantly as the blacklist size scales from 10 to 3,000 speakers. Notably, the Top-1 identification task proves more sensitive to scaling than the Top-S detection task.

In the challenge results, the winning team achieved a 32% improvement in Top-S detection (reducing EER from a 7.52% baseline to 4.25%) and a 46% improvement in Top-1 identification (achieving a 6.03% EER). However, many teams struggled with overfitting; while many outperformed the baseline on development data, only 40% maintained that performance on the unseen test set.

Conclusion and Future Directions

MCE 2018 highlights the significant hurdles of massive-scale speaker detection. While i-vectors trained on large unlabeled datasets proved robust, the lack of raw audio limited the use of modern end-to-end deep learning. Future work aims to provide richer feature sequences that preserve privacy while allowing for more sophisticated analysis to combat channel mismatch and device-switching tactics used by fraudsters to evade detection.

Summary updated

Abstract

The Multi-target Challenge aims to assess how well current speech technology is able to determine whether or not a recorded utterance was spoken by one of a large number of blacklisted speakers. It is a form of multi-target speaker detection based on real-world telephone conversations. Data recordings are generated from call center customer-agent conversations. The task is to measure how accurately one can detect 1) whether a test recording is spoken by a blacklisted speaker, and 2) which specific blacklisted speaker was talking. This paper outlines the challenge and provides its baselines, results, and discussions.
BibTeX
@inproceedings{shon2019mce,
  author = {Suwon Shon and Najim Dehak and Douglas A. Reynolds and James R. Glass},
  title = {{MCE 2018: The 1st Multi-Target Speaker Detection and Identification Challenge Evaluation}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1572},
}
Figure