TitaNet: Neural Model for Speaker Representation with 1D Depth-Wise Separable Convolutions and Global Context

Nithin Rao Koluguri, Taejin Park, Boris Ginsburg

TL;DR

TitaNet is a scalable neural architecture from NVIDIA that redefines how we extract speaker embeddings by combining efficient 1D depth-wise separable convolutions with global context via Squeeze-and-Excitation layers. It achieves state-of-the-art performance in speaker verification (0.68% EER on VoxCeleb1) and diarization while remaining significantly more parameter-efficient than previous benchmarks.

What Problem Does This Paper Address?

In the world of speech processing, "speaker representation" is the art of turning a variable-length audio clip into a fixed-length mathematical vector (often called a t-vector) that uniquely identifies a person's voice. This is critical for two tasks: Speaker Verification (is this person who they say they are?) and Speaker Diarization (who spoke when?).

While models like the x-vector (based on Time Delay Neural Networks) have been the industry standard, they often struggle to capture long-range global context across an entire utterance. Furthermore, many existing systems require complex, multi-stage pipelines involving external models like Probabilistic Linear Discriminant Analysis (PLDA) to compare voices. The authors sought a scalable, end-to-end architecture that captures both local acoustic features and global speaker traits more efficiently.

Methods

TitaNet adapts the architecture of ContextNet (originally designed for Automatic Speech Recognition) for speaker tasks. Its design rests on three pillars:

  1. 1D Depth-wise Separable Convolutions: Instead of standard convolutions, TitaNet uses separable versions that drastically reduce the number of parameters and computational cost while maintaining the ability to model temporal patterns.
  2. Global Context (SE Layers): The model incorporates Squeeze-and-Excitation (SE) modules. These modules use global average pooling to look at the entire sequence at once, allowing the network to "re-weight" local features based on the global context of the utterance.
  3. Attentive Statistics Pooling: To convert the sequence of features into a single vector, the model uses a channel-attention mechanism. This allows the network to focus on the most "speaker-discriminative" parts of the audio.

The model is trained end-to-end using Additive Angular Margin (AAM) loss, which forces the embeddings of the same speaker to be closer together in a hyperspherical space. This allows the system to use simple cosine similarity for comparisons, bypassing the need for a separate PLDA model.

Datasets

The authors trained TitaNet on a massive combined dataset of ~4.8 million utterances from over 16,000 speakers, including:

  • VoxCeleb1 & 2: The gold standard for "in-the-wild" speaker recognition.
  • NIST SRE, Fisher, Switchboard, and LibriSpeech: Diverse datasets covering telephonic and clean speech.

For evaluation, they used the VoxCeleb1 cleaned test trial for verification and several benchmarks for diarization, including AMI (Lapel and MixHeadset), NIST-SRE-2000, and CH109.

Key Insights and Ideas

  • ASR Architectures Translate Well: A key "aha moment" is that architectures optimized for what is being said (ASR) can be successfully adapted to identify who is saying it, provided they are tuned to capture identity-specific traits.
  • Global Context is Crucial: By introducing global average pooling within the SE modules, TitaNet captures speaker characteristics that are consistent across an entire recording, which local-only models (like standard TDNNs) might miss.
  • Efficiency through Width Scaling: The authors found that scaling the "width" (number of channels) of the network is a highly effective way to reduce model size with only a marginal hit to performance.
  • Simplicity in the Backend: By using AAM loss, the researchers proved that a simple cosine similarity check can outperform or match complex statistical backends like PLDA, simplifying the deployment of these models.

Key Results

TitaNet was released in three sizes: Small (6M params), Medium (13.4M), and Large (25.3M).

  • State-of-the-Art Verification: TitaNet-L achieved an Equal Error Rate (EER) of 0.68% on VoxCeleb1, outperforming the popular ECAPA-TDNN (0.69%) and Conformer-based models (2.43%).
  • Superior Diarization: On the AMI-Lapel dataset, TitaNet-M achieved a Diarization Error Rate (DER) of 1.99%, beating ECAPA-TDNN (2.36%) despite being roughly half the size.
  • Parameter Efficiency: TitaNet-S, with only 6 million parameters, achieves results that are competitive with much larger models, making it ideal for edge device applications.

Limitations and Future Work

The authors noted that while larger models significantly improve speaker verification, the gains in diarization are less pronounced. They hypothesize that once the embedding space is "separable enough" for clustering, increasing precision further yields diminishing returns for diarization.

Future work might involve exploring how these embeddings perform in extremely noisy environments or multi-speaker overlapping speech scenarios, which remain a challenge for most diarization systems. The models and code are currently available via the NVIDIA NeMo toolkit.

Summary updated

Abstract

In this paper, we propose TitaNet, a novel neural network architecture for extracting speaker representations. We employ 1D depth-wise separable convolutions with Squeeze-and-Excitation (SE) layers with global context followed by channel attention based statistics pooling layer to map variable-length utterances to a fixed-length embedding (t-vector). TitaNet is a scalable architecture and achieves state-of-the-art performance on speaker verification task with an equal error rate (EER) of 0.68% on the VoxCeleb1 trial file and also on speaker diarization tasks with diarization error rate (DER) of 1.73% on AMI-MixHeadset, 1.99% on AMI-Lapel and 1.11% on CH109. Furthermore, we investigate various sizes of TitaNet and present a light TitaNet-S model with only 6M parameters that achieve near state-of-the-art results in diarization tasks.
BibTeX
@inproceedings{koluguri2022titanet,
  author = {Nithin Rao Koluguri and Taejin Park and Boris Ginsburg},
  title = {{TitaNet: Neural Model for Speaker Representation with 1D Depth-Wise Separable Convolutions and Global Context}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=xfp7WSHKes},
}

Loading related content...

Figure