Benchmarking Speaker Embedders in the Sub-Second Regime for Real-Time Voice AI Agents

Abhishek GautamarXiv 2026 · arXiv

TL;DR

This paper benchmarks state-of-the-art speaker verification models under the extreme constraints of real-time voice AI: sub-second audio chunks and competing background speech. The authors discover that common "best practices"—like denoising audio before processing or truncating audio to fixed lengths—actually degrade performance, and they reveal that roughly half of a standard speaker embedding is redundant for identity verification.

What Problem Does This Paper Address?

Real-time Voice AI agents (like those used in customer service or smart homes) face a "barge-in" problem. While modern denoisers are great at removing static noise like fans or traffic, they often let intelligible background speech (from a TV or a family member) pass through. This leads to "false acceptances," where the AI mistakenly responds to someone who isn't the primary user.

The challenge is twofold:

  1. Latency: To remain "real-time," the system must decide if a speaker is the authorized user within less than a second.
  2. Short Utterances: Speaker verification models typically need several seconds of audio to be accurate. In live conversation, many turns are just "yes," "no," or "okay," providing very little data for the model to work with.

Methods

The researcher evaluated five open-source speaker embedders: TitaNet (Large & Small), ECAPA-TDNN, CAM++, and SpeakerNet.

The study employed a two-pronged evaluation strategy:

  • Controlled Interference Benchmark: To simulate realistic background noise, the author took 400 real customer calls and "injected" a second speaker into the silence intervals at a lower volume (50% amplitude). This allowed for precise measurement of how well models distinguish the target from the background.
  • Production Validation: The models were tested on over 11,000 human-annotated samples from live production traffic to see how they handled real-world channel noise and microphone variability.
  • Ablation Studies: The author tested the impact of two popular denoisers (DeepFilterNet and DTLN) and performed a dimensionality analysis to see which parts of the 192-D TitaNet embedding actually carry useful information.
Fig. 1. Benchmark call construction with a clean 15 s enrollment prefix and attenuated insertions of a second real speaker into silence intervals with 400 ms safety buffers.

Datasets

  • Controlled Benchmark: 400 real customer calls with injected interference (no synthetic speech).
  • Production Set: 11,070 weighted samples from live traffic, including 6,661 target utterances, 1,328 background utterances, and 1,200 noise samples.
  • Pre-training: Models used in the study were pre-trained on standard large-scale datasets like VoxCeleb and 3D-Speaker.

Key Insights and Ideas

  • The Denoising Paradox: Counter-intuitively, denoising audio before embedding it is generally harmful. Denoisers like DeepFilterNet smooth out the fine spectral details (formants) that verification models use to distinguish one voice from another. While the audio sounds "cleaner" to a human, it becomes "homogenized" to the model.
  • Context is King: Truncating audio to a fixed short window (e.g., 700ms) to save time degrades every model. Using the full variable-length segment provided by a Voice Activity Detector (VAD) provides 200–300ms of extra context that significantly stabilizes the speaker embedding.
  • Embedding Redundancy: For TitaNet, the first 92 dimensions of the 192-D embedding contain almost all the discriminative "identity" information. The remaining dimensions largely encode population attributes like gender and language. In a demographically homogeneous user base (e.g., mostly male Hindi speakers), these trailing dimensions add no value and can be safely discarded to save on storage and computation.

Key Results

  • CAM++ was the winner on the controlled benchmark with a 28.69% Equal Error Rate (EER).
  • TitaNet-Small emerged as the production winner on live traffic (30.57% EER), proving that smaller models can sometimes generalize better to real-world channel noise than larger, "tuned" architectures.
  • ECAPA-TDNN showed a massive performance gap, performing well on the benchmark but collapsing to near-chance (51.3% EER) on live production data, highlighting the danger of relying solely on controlled benchmarks for model selection.
Fig. 2. Baseline EER on the raw controlled benchmark. CAM++ leads. TitaNet variants and ECAPA-TDNN sit within 0.21 points of one another, about one point behind CAM++. SpeakerNet is near chance.
Fig. 3. Denoising ablation on the controlled benchmark. Deltas above each pair mark the change in EER after denoising. ECAPA-TDNN with DTLN is the single improving configuration and still does not match raw-audio CAM++.

Limitations and Future Work

  • Demographics: The dimensionality findings are specific to a male-skewed, Hindi-speaking population. The author warns that trailing dimensions might be necessary for more demographically diverse groups.
  • Overlapping Speech: The study focused on background speech in silence intervals. Handling "cocktail party" scenarios where two people speak simultaneously remains a challenge for future work.
  • Single-Chunk Decisions: The EERs (around 30%) are high compared to standard benchmarks because they rely on a single sub-second chunk. The author suggests that aggregating scores across multiple chunks of a call is the next logical step to improve accuracy.

Summary updated

Abstract

Real-time Voice AI agents must separate the active user from intelligible background speech that survives conventional noise suppression. Background sources include television audio, family members, and colleagues. Production denoisers remove non-stationary environmental noise effectively. Background speech shares the spectral characteristics of the target and passes through, causing false acceptances in downstream speaker verification (SV) and automatic speech recognition (ASR). We benchmark open-source speaker embedders TitaNet-Large, TitaNet-Small, ECAPA-TDNN, CAM++, and SpeakerNet under a strict sub-second latency budget. Evaluation combines a controlled benchmark with a weighted human-annotated production set. Our controlled benchmark is built from real calls in which a second real speaker is injected into silence intervals at modulated amplitude. We report three findings. First, denoising before embedding is generally harmful and raises equal error rate (EER) across architectures. Second, truncating audio below the full voice-activity segment degrades every model. Third, roughly half of the TitaNet embedding is redundant for this task. Trailing dimensions encode population attributes such as language and gender. Such attributes carry little discriminative value in a demographically homogeneous user base. We distill these results into deployment guidance. Verification should run on raw audio with full voice-activity chunks. Thresholds should favor precision over F1. Truncated embeddings reduce the cost of similarity search.
BibTeX
@misc{gautam2026benchmarking,
  author = {Abhishek Gautam},
  title = {{Benchmarking Speaker Embedders in the Sub-Second Regime for Real-Time Voice AI Agents}},
  year = {2026},
  eprint = {2611.09516},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09516},
}
Figure