Learning de-identified representations of prosody from raw audio

Jack Weston, Raphael Lenain, Udeepa Meepegama, Emil FristedICML 2021

This paper, presented at ICML 2021 by researchers from Novoic, introduces a novel self-supervised method for learning speech representations that capture prosody (the "how" of speech) while intentionally discarding speaker identity (the "who"). By combining aggressive signal processing with a quantized Transformer architecture, the authors achieve state-of-the-art privacy ("de-identification") without sacrificing the ability to detect complex states like depression or sarcasm.

TL;DR

The authors introduce VQP (Vector Quantized Prosody), a self-supervised model that learns to represent pitch, rhythm, and tempo from raw audio while stripping away speaker-identifiable timbre. It achieves high performance on spoken language understanding tasks while being significantly more "de-identified" than popular baselines like wav2vec 2.0 or TRILL.

What Problem Does This Paper Address?

In speech processing, there is a constant tension between utility and privacy.

  1. Utility: We want models to understand "prosody"—the patterns of stress, intonation, and rhythm that convey emotion, sarcasm, or health indicators (like the "depressive cold" in a voice).
  2. Privacy: Prosody is often entangled with "timbre," the acoustic quality that identifies a specific speaker.

Most existing models for prosody rely on architectural "bottlenecks" (forcing information through a tiny vector) to achieve disentanglement, which is often inefficient. Furthermore, standard speech models (like wav2vec 2.0) are designed for speech recognition; they excel at capturing phonetic detail but are highly identifiable, making them a privacy risk for sensitive applications like medical screening.

Methods

The authors propose a model called VQP that uses "inductive biases"—design choices based on the physics of speech—rather than relying solely on architectural bottlenecks.

  • Aggressive Preprocessing: The raw audio is resampled to 16kHz and then pitch-shifted so every speaker has the same median pitch (150Hz). Crucially, the audio is then downsampled to 500Hz. This retains the fundamental frequency ($F_0$) but discards the high-frequency "formants" that characterize a person's unique vocal tract.
  • Word-Level Slicing: Unlike other models that use fixed-length frames, VQP slices audio into variable-length "audio-words" based on timestamps.
Architecture of the Vector Quantized Prosody (VQP) model, featuring a temporal convolutional network, vector quantization codebooks, and a Transformer encoder.
  • The VQP Architecture:
    • Temporal Convolutional Network (TCN): Extracts features from the 500Hz signal using 1D causal convolutions and residual connections.
  • Product Quantizer: Maps the TCN output to a set of discrete "codebook" vectors. This forces the model to categorize prosodic states (e.g., "high pitch, fast tempo").
  • Transformer Encoder: Contextualizes these discrete "prosody words" across the entire sentence.
  • Self-Supervised Learning: The model is trained using a contrastive loss (similar to BERT or wav2vec 2.0), where it must predict masked segments of the quantized prosody sequence.

Datasets

  • CALC (Colossal Audio-Linguistic Corpus): A massive 7.5-million-word dataset compiled from five sources (including TED-LIUM and AMI) used for self-supervised pretraining.
  • DAMMP Benchmark: A new benchmark introduced in this paper for Depression, Alzheimer’s, Metaphor, MOSEI (sentiment), and Persuasiveness. It standardizes evaluation across five existing datasets (like DAIC-WOZ and ADReSS) where prosody is a key signal.

Key Insights and Ideas

  • The 500Hz Limit: The authors exploit the fact that prosody lives at low frequencies. By capping the input at 500Hz, they create a "physical" privacy barrier that high-resolution models lack.
  • MDL Probing for "Explainability": They use Minimum Description Length (MDL) probing to measure how much information is in their representations. MDL measures the "effort" (in bits) required to extract a feature (like pitch). They found that VQP naturally separates pitch into one part of its codebook and rhythm into another—achieving disentanglement without being explicitly told to do so.
  • Defining De-identifiability: They derive an information-theoretic "de-identification ratio" (DIR). A higher DIR means it is harder for an attacker to link a representation back to a specific speaker's "voiceprint."

Key Results

Performance comparison of VQP against baseline models across the DAMMP benchmark datasets, highlighting VQP's superior De-Identification Ratio (DIR).
  • Privacy: VQP is the most de-identified model tested, achieving a DIR of 1.10. On a speaker verification task with 10 people, an attacker using VQP representations would only identify the correct speaker 1.58% of the time, compared to 37.7% for wav2vec 2.0 and 66.1% for standard x-vectors.
  • Performance: Despite having much less data (due to the 500Hz downsampling), VQP performed best on the DAIC-WOZ (Depression) task (AUC 0.667).
  • Trade-offs: VQP had an average DAMMP score of 0.633, slightly lower than vq-wav2vec (0.687) and TRILL (0.687), but it provided significantly better privacy protection, establishing a new state-of-the-art for privacy-preserving prosody representation.

Limitations and Future Work

  • Short Utterances: VQP struggled on datasets with very short clips (1–20 words), likely because its word-level representations didn't have enough temporal context to pool effectively.
  • Dependency on Alignment: The model currently requires word-level timestamps (from ASR or forced alignment) to slice the audio, which adds a dependency on a separate phonetic model.
  • Laughter and Non-speech: The authors noted that VQP ignores non-speech sounds (like laughter) that occur after a word ends, which might be why it underperformed on the MUStARD (sarcasm) dataset. Future versions could benefit from including these "inter-word" acoustic cues.

Summary updated

BibTeX
@inproceedings{weston2021learning,
  author = {Jack Weston and Raphael Lenain and Udeepa Meepegama and Emil Fristed},
  title = {{Learning de-identified representations of prosody from raw audio}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure