Historically, audio processing models relied on one-dimensional waveforms or treated two-dimensional spectrograms with Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). While effective for local patterns, these architectures struggle with the non-local nature of sound. Acoustic signals contain long-range dependencies: a single harmonic overtonal structure spans wide frequency bands simultaneously, and intermittent sound events—like a bird call or a respiratory wheeze—can repeat seconds apart. CNNs are constrained by local receptive fields, failing to connect these distant components until the deepest layers of the network.
To overcome these limitations, a powerful paradigm shift borrows an insight from computer vision: treating a log-Mel spectrogram not as a continuous canvas for local convolutions, but as a structured sequence of discrete, uniform two-dimensional patches. By tokenizing audio in this manner, sequence models can capture global relationships across both time and frequency axes from the very first layer.
The Mechanics of Spectrogram Patchification
The pipeline for patch-based audio representation follows a clean, sequential workflow:
- Time-Frequency Conversion: The raw, one-dimensional audio waveform is transformed into a two-dimensional log-Mel spectrogram, mapping energy across time and frequency.
- Grid Decomposition: Instead of scanning the spectrogram sequentially, the model divides the image-like representation into a grid of small, fixed-size patches (e.g., $16 \times 16$ squares).
- Linear Projection: Each patch is flattened into a vector and projected through a linear layer into a high-dimensional embedding space, effectively turning a piece of the spectrogram into a discrete token.
- Positional Encoding: Because sequence backbones are permutation-invariant, learnable positional encodings are added to the patch embeddings to preserve their original coordinates in time and frequency.
Raw Audio Waveform
│
▼
┌──────────────────────────────┐
│ Log-Mel Spectrogram │ (2D Time-Frequency Representation)
└──────────────────────────────┘
│
▼
┌───┬───┬───┬───┐
├───┼───┼───┼───┤ (Decomposed into a grid of uniform patches)
└───┴───┴───┴───┘
│
▼
[Patch 1] [Patch 2] [Patch 3] ... (Flattened, projected, and position-encoded)
│
▼
┌──────────────────────────────┐
│ Sequence Modeling Engine │ (Transformer Encoder or State Space Model)
└──────────────────────────────┘
The evolution of this tokenization strategy moved from one-dimensional slicing to full two-dimensional grids. Early implementations, such as the Keyword Transformer introduced by Berg et al., divided the spectrogram into non-overlapping segments purely along the time axis. While effective for localized tasks like keyword spotting, this approach ignored explicit frequency-axis boundaries.
To capture the full geometry of sound, subsequent architectures transitioned to two-dimensional grid decomposition. The Audio Spectrogram Transformer (AST), developed by Gong et al., and the Audio Captioning Transformer, introduced by Mei et al., demonstrated that a 2D patch grid allows the model to process time and frequency coordinates symmetrically, enabling a pure, convolution-free architecture to outperform complex hybrid models.
Why It Works: Global Context and Receptive Fields
The primary reason patch-based tokenization succeeds is that it pairs the representation with a global sequence modeling engine, such as a Transformer encoder. When self-attention is applied to a sequence of spectrogram patches, every patch can interact with every other patch regardless of their distance in time or frequency.
As noted by Mouchtaris et al., this global context capture allows models to weigh the importance of different acoustic events simultaneously across an entire utterance. In practical terms, this solves several classic audio processing challenges:
- Polyphony and Overlapping Sounds: In complex environments with overlapping audio sources, local kernels often conflate mixed signals. Angulo et al. demonstrated that patch-based self-attention allows urban sound monitoring systems to untangle highly polyphonic environments by correlating distant time-frequency components.
- Harmonic Structures: Symmetrical encoder-decoder architectures leverage global attention to track repetitive or harmonic structures that span large vertical distances on a spectrogram. Kumar et al. utilized this capability for bird sound denoising, successfully separating intricate avian vocalizations from non-stationary environmental noise like wind and rain.
- Intermittent Events: For non-periodic acoustic events separated by long intervals, a global receptive field ensures that the network retains historical context. Bae et al. showed that this property is essential for identifying sporadic clinical anomalies in respiratory sounds.
Self-Supervised Learning via Patch Manipulation
Decomposing a spectrogram into discrete tokens unlocks powerful self-supervised learning frameworks originally designed for natural language processing and computer vision. Rather than relying on massive labeled datasets or cross-modality pretraining from image datasets, models can learn the underlying physics of sound by manipulating patches.
Masked Spectrogram Prediction
In this paradigm, a large percentage of the 2D spectrogram patches are randomly masked before being fed into the encoder. The model is then tasked with reconstructing the missing time-frequency pieces.
Gong et al. introduced this framework with SSAST, proving that masking and predicting 2D patches allows a model to learn complex relationships across both axes from unlabeled data alone. Similarly, the MaskSpec framework by Chong et al. confirmed that direct patch reconstruction effectively closes the "domain gap" between computer vision and audio processing, matching the performance of models pre-trained on massive image repositories.
Patch-Mix Augmentation
Because patches function as independent tokens, data augmentation can be performed directly at the sub-sample level. Bae et al. developed "Patch-Mix Contrastive Learning," a technique that randomly swaps patches between different audio samples. By forcing the model to recognize these intra-sample mixtures through a specialized contrastive loss, the network develops a more robust understanding of overlapping, intermittent acoustic events.
Decoupling the Representation from the Backbone
While patchification was popularized by the Audio Spectrogram Transformer, the representational strategy itself is backbone-agnostic. The core innovation is treating the time-frequency grid as a sequence of discrete tokens, meaning the downstream sequence engine can be swapped to optimize for different performance trade-offs.
A notable limitation of the Transformer architecture is the quadratic computational complexity ($O(N^2)$) of self-attention, which becomes prohibitive as audio recordings grow longer. To address this, researchers have integrated patch-based tokenization with State Space Models (SSMs) like Mamba, which scale linearly ($O(N)$).
Shams et al. and Senocak et al. independently demonstrated that log-Mel spectrograms can be partitioned into fixed-size patches, flattened, augmented with positional encodings, and fed into a bidirectional Mamba encoder. This approach retains the global context and cross-axis modeling strengths of patch-based representations while delivering massive improvements in execution speed and memory efficiency.
Universal Application Across Audio Domains
The versatility of treating audio as a sequence of patches is reflected in its widespread adoption across highly diverse domains. The same underlying mechanism of patch projection and sequence modeling has driven state-of-the-art results in:
- Speech and Semantics: Predicting non-intrusive speech quality dimensions directly from degraded signals (Wardah et al.).
- Medical Diagnostics: Analyzing snoring audio to predict sleep body and head positions with high accuracy as a non-invasive diagnostic tool (Xiao et al.).
- Music Information Retrieval: Capturing long-range dependencies across time and frequency axes to enable cross-cultural music tagging and transfer learning between Western and Eastern musical traditions (Papaioannou et al.).
By converting continuous acoustic signals into structured sequences of visual-like patches, this methodology unifies audio processing under the same foundational sequence-modeling paradigms driving modern artificial intelligence.
Key Foundational Work
Audio Captioning Transformer (2021) — contributed 3 concept elements to this direction.
Keyword Transformer: A Self-Attention Model for Keyword Spotting (2021) — contributed 2 concept elements to this direction.
AST: Audio Spectrogram Transformer (2021) — contributed 1 concept elements to this direction.