Abstract
Self-supervised learning has been a powerful approach for learning meaningful representations from unlabeled data across various domains, reducing the reliance on large labeled datasets. Inspired by BERT's success in capturing deep bidirectional contexts in natural language processing, similar frameworks have been adapted to other modalities such as audio, with models like BEATs extending the bidirectional training paradigm to audio signals using vector quantization (VQ). However, these frameworks face challenges, notably their dependence on a single codebook for quantization, which may not capture the complex, multifaceted nature of signals. In addition, inefficiencies in codebook utilization lead to underutilized code vectors. To address these limitations, we introduce BRIDLE (Bidirectional Residual Quantization Interleaved Discrete Learning Encoder), a self-supervised encoder pretraining framework that incorporates residual quantization (RQ) into the bidirectional training process, and is generalized for pretraining with audio, image, and video. Using multiple hierarchical codebooks, RQ enables fine-grained discretization in the latent space, enhancing representation quality. BRIDLE involves an interleaved training procedure between the encoder and tokenizer. We evaluate BRIDLE on audio understanding tasks using classification benchmarks, achieving state-of-the-art results, and demonstrate competitive performance on image classification and video classification tasks, showing consistent improvements over traditional VQ methods in downstream performance.
Introduction
Self-supervised learning (SSL) has revolutionized AI by enabling models to learn from massive unlabeled datasets. A popular paradigm, inspired by BERT, involves "tokenizing" continuous data (like audio or images) into discrete codes and training an encoder to predict masked portions of these sequences. However, existing frameworks often rely on Vector Quantization (VQ) with a single codebook, which presents two major hurdles:
- Complexity Gap: A single codebook often lacks the resolution to capture the intricate, multi-dimensional details of complex signals like high-fidelity audio or video.
- Codebook Inefficiency: Training often suffers from "codebook collapse," where only a small fraction of available code vectors are utilized, leading to suboptimal representations and wasted computational capacity.
Methodology: The BRIDLE Framework
To overcome these challenges, we propose BRIDLE (Bidirectional Residual Quantization Interleaved Discrete Learning Encoder). The architecture consists of four primary components: a Main Encoder, a Tokenizer (containing multiple codebooks), a Main Decoder, and a Tokenizer Estimator.
Interleaved Training Procedure
The core of BRIDLE is its interleaved training strategy, which alternates between optimizing the encoder and the tokenizer to create a virtuous cycle of representation learning.
- Encoder Phase: The encoder learns to predict discrete tokens generated by the tokenizer for masked inputs. In this phase, the tokenizer is frozen while the encoder and decoder are updated using a cross-entropy loss.
- Tokenizer Phase: The tokenizer acts as a student, learning to align its outputs with the continuous embeddings produced by the "teacher" encoder. Here, the encoder is frozen, and the tokenizer components are updated using cosine similarity and codebook losses.
Residual Quantization (RQ)
Unlike standard VQ, BRIDLE employs Residual Quantization (RQ). Instead of mapping a vector to a single code, RQ uses a hierarchy of codebooks. The first codebook captures the coarse structure, and subsequent codebooks quantize the "residual" (the error) left by the previous stage. This hierarchical approach allows for a much finer discretization of the latent space.
To maintain codebook "health," we implement several optimization techniques:
- EMA Updates: Exponential Moving Averages for stable codebook updates.
- k-means Initialization: Aligning codebooks with the data distribution from the first batch to ensure they are well-aligned from the start.
- Code Resetting: Re-initializing underutilized codes to prevent "dead" neurons and encourage exploration.
Experimental Setup
The framework was rigorously evaluated across three modalities to demonstrate its generalizability:
- Audio: Pretrained on AudioSet (2M clips); evaluated on AudioSet-2M, AudioSet-20K, and ESC-50.
- Image: Pretrained and evaluated on ImageNet-1K (1.28M images).
- Video: Pretrained and evaluated on Kinetics-400 (~240,000 action clips).
Results and Discussion
BRIDLE consistently outperformed traditional VQ-based methods across all tested modalities. On AudioSet-2M, BRIDLE achieved a mean Average Precision (mAP) of 47.99, surpassing previous benchmarks. For vision tasks, fine-tuning accuracy on ImageNet-1K reached 81.10% (Top-1), compared to 80.26% for the VQ baseline, while Kinetics-400 performance rose to 72.90%.
Key Insights
- Hierarchy Matters: The shift from VQ to RQ is akin to moving from a low-resolution grid to a multi-scale map, allowing the model to represent subtle nuances.
- Interleaving as Self-Distillation: The bidirectional training creates a cycle where the tokenizer provides better targets for the encoder, which in turn provides better features for the tokenizer to discretize.
- Cross-Modal Generalization: One of the most significant findings is that the same fundamental framework works across 1D audio, 2D images, and 3D video with minimal modality-specific adjustments.
Conclusion and Future Work
While BRIDLE shows significant promise, joint training (simultaneous optimization) of the encoder and tokenizer proved difficult to tune for complex RQ models. Additionally, scaling beyond a certain number of codes led to diminishing returns. Future work will explore adaptive codebook sizes and cross-modal applications such as audio-visual speech recognition.
Summary updated