wav2tok: Deep Sequence Tokenizer for Audio Retrieval

Adhiraj Banerjee, Vipul AroraICLR 2023

TL;DR

wav2tok is a self-supervised framework that encodes continuous audio signals into concise sequences of discrete tokens. By training on pairs of similar audio (like a hummed tune and its studio version), it learns to extract "semantic" units that enable lightning-fast and highly accurate audio retrieval using simple string matching.

What Problem Does This Paper Address?

Searching through audio is computationally expensive. Traditionally, researchers use Dynamic Time Warping (DTW) to compare two audio signals. While effective, DTW is slow because it operates on high-dimensional, continuous feature vectors.

An alternative is to use Automatic Speech Recognition (ASR) to turn audio into text (tokens), which can then be searched instantly. However, ASR requires massive amounts of labeled data (transcripts) and is strictly tied to a specific language. What if you want to search for a non-linguistic sound, like a cough, or a melody where no "alphabet" exists?

The authors address the challenge of unsupervised sequence retrieval: learning to map any audio—speech or music—into a discrete "alphabet" of tokens without needing manual transcriptions or domain-specific experts.

Methods

The core of wav2tok is a neural network architecture designed to compress audio into a "pseudo-orthography" (a written form for sounds).

wav2tok Model Architecture and Training Pipeline
  1. The Encoder: A 2-layer Bidirectional LSTM (BiLSTM) processes raw audio features (like spectrograms) into continuous latent representations.
  2. The Quantizer: These continuous vectors are mapped to the nearest "codebook" vector using a $K$-means strategy. Each vector in the codebook represents a discrete token (e.g., Token #14).
  3. The Compressor: To make the sequences concise, the model deletes consecutive repetitions of the same token (e.g., [1, 1, 2, 2, 2, 1] becomes [1, 2, 1]).
  4. The Training Strategy: This is the "secret sauce." The model is trained on pairs of similar audio. For music, this might be two different people humming the same song. The training uses two losses:
  5. Contrastive Loss ($L_m$): Ensures that the representations for similar sounds are pushed closer to their assigned codebook tokens and further from others.
  6. CTC Loss ($L_{ctc}$): Borrowed from speech recognition, this loss constrains both audio clips in a pair to map to the same sequence of tokens, even if their durations differ.
  7. EM-like Optimization: The authors use an Expectation-Maximization approach. They periodically run offline $K$-means clustering to update the codebook (the "E-step") and use gradient descent to train the BiLSTM (the "M-step"). This prevents "codebook collapse," where the model tries to map every sound to the same single token.

Datasets

The model was evaluated on two distinct tasks:

  • Music (Query by Humming): Using the MIR-QbSH dataset (4,431 humming recordings of 48 songs).
  • Speech (Spoken Term Detection): Using the TIMIT dataset (6,300 utterances) and a 100-hour subset of LibriSpeech for larger-scale pre-training.

Key Insights and Ideas

  • Melody as Language: The model treats a melody like a language. By training on pairs, it learns that the "singer's identity" is noise, but the "sequence of notes" is the semantic content.
  • Efficiency via Discretization: By converting audio to tokens, search becomes a string-matching problem. The authors show that searching over these tokens is two orders of magnitude faster than performing DTW on continuous features.
  • Small is Beautiful: Unlike massive models like wav2vec 2.0 (which can have 300M+ parameters), wav2tok uses a relatively tiny BiLSTM (3.6M parameters). Despite its size, it outperforms the giants on these specific retrieval tasks because its pairwise training is specifically tuned for similarity.

Key Results

  • Superior Retrieval: On the Query by Humming task, wav2tok achieved a Mean Reciprocal Rank (MRR) of 0.84, significantly outperforming standard MIDI-based matching (0.75) and triplet-loss baselines (0.50).
  • Robustness: The tokens learned are invariant to pitch shifts and time stretching. When a user hums off-key or at a different speed, wav2tok still generates the correct token sequence.
  • Speech Search: In Spoken Term Detection, wav2tok outperformed fine-tuned versions of wav2vec 2.0, proving that its "tokenization" is more effective for finding specific words in a stream of audio than general-purpose phonetic models.

Limitations and Future Work

  • Ordered Alignment: The method assumes that the query and the target have an ordered alignment (the sounds happen in the same relative order). It wouldn't work for "shuffled" audio or finding a clip where the middle section is missing.
  • Equidistant Assumption: The current matching algorithm assumes all tokens are equidistant in "semantic space." The authors suggest that future work could explore the metric space of these tokens (e.g., is Token A "closer" to Token B than Token C?).
  • Domain Specificity: While the framework is general, the tokens learned for music don't necessarily transfer to speech. A separate model must be trained for different domains of sound.

Summary updated

BibTeX
@inproceedings{banerjee2023wav,
  author = {Adhiraj Banerjee and Vipul Arora},
  title = {{wav2tok: Deep Sequence Tokenizer for Audio Retrieval}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
}
Figure