Introduction
Traditional neural audio compression typically utilizes Vector Quantized Variational Auto-encoders (VQ-VAEs), which represent audio as a dense matrix of integers. While effective, this approach is inherently rigid, allocating a fixed number of bits regardless of the audio's complexity or silence. In contrast, biological systems communicate via sparse, energy-efficient "spikes." This research introduces "Spiking Music," a neural audio compression method that replaces traditional integer-based bottlenecks with sparse binary events. By forcing extreme sparsity, the model achieves competitive compression at 3kbps and spontaneously learns to map specific "neurons" to individual musical notes without supervision, mirroring the efficiency of formats like MIDI.
Methodology
The Spiking Music framework utilizes a standard neural compression architecture—specifically a SeaNet encoder and a diffusion-based decoder—but introduces a radical change at the bottleneck.
Key Technical Components
- Binary Bottleneck: The latent space is transformed into a binary matrix $z \in \{0, 1\}^{N,T_z}$. To maintain differentiability during training, the authors employ a Heaviside step function with a surrogate gradient.
- Sparsity Pressure: An auxiliary loss function penalizes the number of active events (1s in the matrix), incentivizing the model to find the most parsimonious representation.
- Sparse Storage Logic: The authors demonstrate that for sufficiently sparse matrices, storing event indices (using Coordinate List or Compressed Time formats) is more efficient than dense storage. This allows the bit rate to scale with audio complexity rather than time alone, moving from $O(T_z)$ to $O(\log_2(T_z))$ scaling.
- Variable Sparsity (\mu-SPARSE): To avoid representation collapse, a variant was developed where the target bit rate can be adjusted via a "compression prompt" integer.
Experimental Setup
The models were trained and evaluated on the MAESTRO dataset, which contains over 170 hours of virtuosic piano performances. This dataset is uniquely suited for this task as it provides high-quality audio aligned with precise MIDI data, allowing researchers to verify if the learned "spikes" correspond to actual musical events (keystrokes and pedal actions).
Results and Discussion
Perceptual Quality and Bitrate
The Spiking Music models achieved competitive reconstruction quality at approximately 3 kbps. In MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) listening tests, the "FREE" spiking model (binary but not strictly sparse) often outperformed the Residual Vector Quantization (RVQ) baseline.
Emergent Selectivity
A significant finding was the emergence of selectivity: without explicit supervision, specific binary units began to synchronize with specific MIDI note onsets. This suggests the model discovered the underlying "physics" of the piano to optimize compression. Cross-correlation analysis showed a sharp peak at lag zero between learned spikes and MIDI onsets.
Conclusion
Spiking Music demonstrates that event-based encoding is a viable and theoretically superior approach for low-bitrate compression of sparse signals like music. While training sparse models remains challenging due to potential representation collapse, the emergent selectivity and sub-linear scaling properties offer a promising path toward more efficient, interpretable audio codecs. Future work may explore continuous-time models to further enhance temporal precision and compression ratios.
Summary updated