This paper, presented at ICML 2021 by researchers from Novoic, introduces a novel self-supervised method for learning speech representations that capture prosody (the "how" of speech) while intentionally discarding speaker identity (the "who"). By combining aggressive signal processing with a quantized Transformer architecture, the authors achieve state-of-the-art privacy ("de-identification") without sacrificing the ability to detect complex states like depression or sarcasm.
TL;DR
The authors introduce VQP (Vector Quantized Prosody), a self-supervised model that learns to represent pitch, rhythm, and tempo from raw audio while stripping away speaker-identifiable timbre. It achieves high performance on spoken language understanding tasks while being significantly more "de-identified" than popular baselines like wav2vec 2.0 or TRILL.
What Problem Does This Paper Address?
In speech processing, there is a constant tension between utility and privacy.
- Utility: We want models to understand "prosody"—the patterns of stress, intonation, and rhythm that convey emotion, sarcasm, or health indicators (like the "depressive cold" in a voice).
- Privacy: Prosody is often entangled with "timbre," the acoustic quality that identifies a specific speaker.
Most existing models for prosody rely on architectural "bottlenecks" (forcing information through a tiny vector) to achieve disentanglement, which is often inefficient. Furthermore, standard speech models (like wav2vec 2.0) are designed for speech recognition; they excel at capturing phonetic detail but are highly identifiable, making them a privacy risk for sensitive applications like medical screening.
Methods
The authors propose a model called VQP that uses "inductive biases"—design choices based on the physics of speech—rather than relying solely on architectural bottlenecks.
- Aggressive Preprocessing: The raw audio is resampled to 16kHz and then pitch-shifted so every speaker has the same median pitch (150Hz). Crucially, the audio is then downsampled to 500Hz. This retains the fundamental frequency ($F_0$) but discards the high-frequency "formants" that characterize a person's unique vocal tract.
- Word-Level Slicing: Unlike other models that use fixed-length frames, VQP slices audio into variable-length "audio-words" based on timestamps.
- The VQP Architecture:
- Temporal Convolutional Network (TCN): Extracts features from the 500Hz signal using 1D causal convolutions and residual connections.
- Product Quantizer: Maps the TCN output to a set of discrete "codebook" vectors. This forces the model to categorize prosodic states (e.g., "high pitch, fast tempo").
- Transformer Encoder: Contextualizes these discrete "prosody words" across the entire sentence.
- Self-Supervised Learning: The model is trained using a contrastive loss (similar to BERT or wav2vec 2.0), where it must predict masked segments of the quantized prosody sequence.
Datasets
- CALC (Colossal Audio-Linguistic Corpus): A massive 7.5-million-word dataset compiled from five sources (including TED-LIUM and AMI) used for self-supervised pretraining.
- DAMMP Benchmark: A new benchmark introduced in this paper for Depression, Alzheimer’s, Metaphor, MOSEI (sentiment), and Persuasiveness. It standardizes evaluation across five existing datasets (like DAIC-WOZ and ADReSS) where prosody is a key signal.
Key Insights and Ideas
- The 500Hz Limit: The authors exploit the fact that prosody lives at low frequencies. By capping the input at 500Hz, they create a "physical" privacy barrier that high-resolution models lack.
- MDL Probing for "Explainability": They use Minimum Description Length (MDL) probing to measure how much information is in their representations. MDL measures the "effort" (in bits) required to extract a feature (like pitch). They found that VQP naturally separates pitch into one part of its codebook and rhythm into another—achieving disentanglement without being explicitly told to do so.
- Defining De-identifiability: They derive an information-theoretic "de-identification ratio" (DIR). A higher DIR means it is harder for an attacker to link a representation back to a specific speaker's "voiceprint."
Key Results
- Privacy: VQP is the most de-identified model tested, achieving a DIR of 1.10. On a speaker verification task with 10 people, an attacker using VQP representations would only identify the correct speaker 1.58% of the time, compared to 37.7% for wav2vec 2.0 and 66.1% for standard x-vectors.
- Performance: Despite having much less data (due to the 500Hz downsampling), VQP performed best on the DAIC-WOZ (Depression) task (AUC 0.667).
- Trade-offs: VQP had an average DAMMP score of 0.633, slightly lower than vq-wav2vec (0.687) and TRILL (0.687), but it provided significantly better privacy protection, establishing a new state-of-the-art for privacy-preserving prosody representation.
Limitations and Future Work
- Short Utterances: VQP struggled on datasets with very short clips (1–20 words), likely because its word-level representations didn't have enough temporal context to pool effectively.
- Dependency on Alignment: The model currently requires word-level timestamps (from ASR or forced alignment) to slice the audio, which adds a dependency on a separate phonetic model.
- Laughter and Non-speech: The authors noted that VQP ignores non-speech sounds (like laughter) that occur after a word ends, which might be why it underperformed on the MUStARD (sarcasm) dataset. Future versions could benefit from including these "inter-word" acoustic cues.
Summary updated