Spiking Music: Audio Compression with Event Based Auto-encoders

Martim Lisboa, Guillaume BellecarXiv 2024 · DOI · arXiv

Introduction

Traditional neural audio compression typically utilizes Vector Quantized Variational Auto-encoders (VQ-VAEs), which represent audio as a dense matrix of integers. While effective, this approach is inherently rigid, allocating a fixed number of bits regardless of the audio's complexity or silence. In contrast, biological systems communicate via sparse, energy-efficient "spikes." This research introduces "Spiking Music," a neural audio compression method that replaces traditional integer-based bottlenecks with sparse binary events. By forcing extreme sparsity, the model achieves competitive compression at 3kbps and spontaneously learns to map specific "neurons" to individual musical notes without supervision, mirroring the efficiency of formats like MIDI.

Methodology

The Spiking Music framework utilizes a standard neural compression architecture—specifically a SeaNet encoder and a diffusion-based decoder—but introduces a radical change at the bottleneck.

Architecture of the Spiking Music auto-encoder, showing the binary quantizer bottleneck and the integration of compression prompts.

Key Technical Components

  • Binary Bottleneck: The latent space is transformed into a binary matrix $z \in \{0, 1\}^{N,T_z}$. To maintain differentiability during training, the authors employ a Heaviside step function with a surrogate gradient.
  • Sparsity Pressure: An auxiliary loss function penalizes the number of active events (1s in the matrix), incentivizing the model to find the most parsimonious representation.
  • Sparse Storage Logic: The authors demonstrate that for sufficiently sparse matrices, storing event indices (using Coordinate List or Compressed Time formats) is more efficient than dense storage. This allows the bit rate to scale with audio complexity rather than time alone, moving from $O(T_z)$ to $O(\log_2(T_z))$ scaling.
  • Variable Sparsity (\mu-SPARSE): To avoid representation collapse, a variant was developed where the target bit rate can be adjusted via a "compression prompt" integer.

Experimental Setup

The models were trained and evaluated on the MAESTRO dataset, which contains over 170 hours of virtuosic piano performances. This dataset is uniquely suited for this task as it provides high-quality audio aligned with precise MIDI data, allowing researchers to verify if the learned "spikes" correspond to actual musical events (keystrokes and pedal actions).

Results and Discussion

Perceptual Quality and Bitrate

The Spiking Music models achieved competitive reconstruction quality at approximately 3 kbps. In MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) listening tests, the "FREE" spiking model (binary but not strictly sparse) often outperformed the Residual Vector Quantization (RVQ) baseline.

Average perceptual scores versus bit rate for various models, including the SPARSE and FREE variants.

Emergent Selectivity

A significant finding was the emergence of selectivity: without explicit supervision, specific binary units began to synchronize with specific MIDI note onsets. This suggests the model discovered the underlying "physics" of the piano to optimize compression. Cross-correlation analysis showed a sharp peak at lag zero between learned spikes and MIDI onsets.

Conclusion

Spiking Music demonstrates that event-based encoding is a viable and theoretically superior approach for low-bitrate compression of sparse signals like music. While training sparse models remains challenging due to potential representation collapse, the emergent selectivity and sub-linear scaling properties offer a promising path toward more efficient, interpretable audio codecs. Future work may explore continuous-time models to further enhance temporal precision and compression ratios.

Summary updated

Abstract

Neurons in the brain communicate information via punctual events called spikes. The timing of spikes is thought to carry rich information, but it is not clear how to leverage this in digital systems. We demonstrate that event-based encoding is efficient for audio compression. To build this event-based representation we use a deep binary auto-encoder, and under high sparsity pressure, the model enters a regime where the binary event matrix is stored more efficiently with sparse matrix storage algorithms. We test this on the large MAESTRO dataset of piano recordings against vector quantized auto-encoders. Not only does our "Spiking Music compression" algorithm achieve a competitive compression/reconstruction trade-off, but selectivity and synchrony between encoded events and piano key strikes emerge without supervision in the sparse regime.
BibTeX
@misc{lisboa2024spiking,
  author = {Martim Lisboa and Guillaume Bellec},
  title = {{Spiking Music: Audio Compression with Event Based Auto-encoders}},
  year = {2024},
  eprint = {2402.01571},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.01571},
}
Figure