BAT: Better Audio Transformer Guided by Convex Gated Probing

Houtan Ghaffari, Lukas Rauch, Christoph Scholz, Paul Devos

Abstract

Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as fine-tuning may misrepresent their inherent quality. In contrast, audio SSL models still rely on fine-tuning because simple probing fails to unlock their full potential and alters their rankings when competing for SOTA on AudioSet. Hence, a robust and efficient probing mechanism is required to guide the trajectory of audio SSL towards reliable and reproducible methods. We introduce Convex Gated Probing (CGP), a prototype-based method that drastically closes the gap between fine-tuning and probing in audio. CGP efficiently utilizes all frozen layers via a gating mechanism and exposes the location of latent task-relevant information. Guided by CGP, we rework the entire SSL pipeline of current SOTA audio models that use legacy implementations of prior SSL methods. By refining data preprocessing, model architecture, and pre-training recipe, we introduce Better Audio Transformer (BAT), and establish new SOTA on audio benchmarks.

Introduction

In Self-Supervised Learning (SSL), evaluating the quality of internal representations is crucial. In computer vision, "probing"—training a small linear layer on top of frozen features—is the standard. However, in the audio domain, simple linear probes often yield poor results, lagging significantly behind full fine-tuning. This discrepancy forces researchers to rely on fine-tuning for evaluation, which is computationally expensive, sensitive to hyperparameters, and often obscures the true quality of the learned representations.

To address this, the authors introduce Convex Gated Probing (CGP), a novel evaluation method that makes frozen-feature probing a viable alternative to fine-tuning. By using CGP as a diagnostic tool, they identify bottlenecks in existing state-of-the-art (SOTA) pipelines and develop the Better Audio Transformer (BAT), setting new performance records across major audio benchmarks while ensuring full reproducibility.

Methods

Convex Gated Probing (CGP)

Instead of just looking at the last layer of a transformer, CGP uses a learnable gating mechanism to aggregate information across all layers. The process involves projecting features into a compact "prototype space" (comparing embeddings to learned class-representative vectors) and using a soft-max gate to automatically find where the most task-relevant information lives. This drastically closes the gap between probing and fine-tuning.

Overview of the Convex Gated Probing (CGP) architecture, showing the flow from gated attention to similarity-based classification.

Better Audio Transformer (BAT)

Using CGP to "see" inside existing models, the authors modernized the standard audio SSL pipeline:

  • Better Preprocessing: Legacy speech-centric pipelines were replaced with a modern TorchAudio implementation using decibel-scale log-compression and local min-max normalization.
  • Gated Attention: A sigmoid gating mechanism was added to the attention blocks to prevent "attention sinks" and improve the quality of training targets.
  • Expressive Decoder: The typical lightweight CNN decoder was swapped for a more powerful 6-layer ViT decoder. This "offloads" the reconstruction task from the encoder, allowing it to focus on learning high-level semantic features.

Key Insights and Results

The "Later-is-Better" Shift

By using CGP, the authors discovered that in older models (like EAT), useful semantic information actually peaks in the middle layers and degrades toward the end because the final layers are too busy trying to reconstruct raw spectrograms. By using a heavier decoder in BAT, they successfully pushed the "semantic peak" to the very last layer, making the final embeddings much more useful for downstream tasks.

Performance Benchmarks

BAT establishes a new SOTA across the board:

  • AudioSet-2M: Achieved 48.55 mAP, outperforming the reproduced EAT and SSLAM baselines.
  • Closing the Probing Gap: On AudioSet-20k, CGP performance is within 3.8 points of full fine-tuning. For context, previous linear probes were often 20+ points behind.
  • Zero-Shot Robustness: On the ESC-50 dataset, BAT using the CGP probe actually outperformed full fine-tuning of previous models.
Performance (mAP) of SSLAM and EAT models as a function of the number of prototypes used in the CGP probe.

As shown in the figure above, the performance of the CGP probe scales effectively with the number of prototypes, demonstrating the efficiency of the prototype-based approach in capturing complex audio features.

Conclusion

BAT and CGP represent a significant step toward more reliable and interpretable audio SSL. By providing a robust probing mechanism, this work enables researchers to diagnose model behavior more effectively. The refinements in preprocessing and architecture demonstrate that modernization and transparency can lead to substantial performance gains without increasing model complexity.

Summary updated

Abstract

Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as fine-tuning may misrepresent their inherent quality. In contrast, audio SSL models still rely on fine-tuning because simple probing fails to unlock their full potential and alters their rankings when competing for SOTA on AudioSet. Hence, a robust and efficient probing mechanism is required to guide the trajectory of audio SSL towards reliable and reproducible methods. We introduce Convex Gated Probing (CGP), a prototype-based method that drastically closes the gap between fine-tuning and probing in audio. CGP efficiently utilizes all frozen layers via a gating mechanism and exposes the location of latent task-relevant information. Guided by CGP, we rework the entire SSL pipeline of current SOTA audio models that use legacy implementations of prior SSL methods. By refining data preprocessing, model architecture, and pre-training recipe, we introduce Better Audio Transformer (BAT), and establish new SOTA on audio benchmarks.
BibTeX
@inproceedings{ghaffari2026bat,
  author = {Houtan Ghaffari and Lukas Rauch and Christoph Scholz and Paul Devos},
  title = {{BAT: Better Audio Transformer Guided by Convex Gated Probing}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=9DfsHAFE5l},
}
Figure