Binary Sparse Codes: Where are we on Energy Efficient Representations?

Jose Miguel Ramos, Luis SaCouto, Andreas WichertarXiv 2026 · arXiv

TL;DR

This paper introduces a rigorous evaluation framework to measure the quality of binary sparse codes—representations that mimic the brain's energy-efficient communication. By comparing deep learning, mathematical, and biologically plausible models, the authors reveal a fundamental trade-off between extreme sparsity and information preservation, identifying "Competitive Sparse Codes" as a promising path toward brain-like efficiency.

What Problem Does This Paper Address?

Modern AI is an energy glutton. While the human brain operates on roughly 20 watts, deep learning models require massive power budgets. The brain achieves its efficiency through sparse coding: only a tiny fraction of neurons fire at any given time.

Artificial systems are trying to replicate this using binary sparse codes (vectors of 0s and 1s where 1s are rare). However, the field faces a "measurement gap." While it is easy to measure how sparse a code is, there is no agreed-upon way to measure information preservation—how well the code actually represents the input data. Without a standard metric, researchers cannot fairly compare different methods or understand the inherent trade-offs between saving energy and keeping data meaningful.

Methods

The authors propose a set of metrics to formalize information preservation as similarity preservation (ensuring similar inputs result in similar codes). They then use these metrics to benchmark four distinct paradigms:

  1. Dictionary Learning (DL): A classical mathematical approach that learns "atoms" to represent data. Since it produces continuous values, the authors applied a k-winners-take-all (k-WTA) step to force it into a binary format.
  2. Gumbel Autoencoder (GAE): A deep learning approach trained via backpropagation, using a Gumbel-Softmax trick to handle the discrete nature of binary bits.
  3. Competitive Sparse Codes (CSC): A biologically plausible model that uses local Hebbian learning rules ("cells that fire together, wire together") rather than global backpropagation.
  4. Random Projections (RP): A learning-free baseline that projects data into high-dimensional space using a fixed random matrix.

To measure Sparsity, they use Normalized Shannon Entropy. In binary codes, the number of active bits is often fixed, so "quality" depends on how evenly those bits are used across the whole dataset. High entropy means the system uses its entire "vocabulary" efficiently.

Datasets

The study uses the MNIST dataset of handwritten digits.

  • Preprocessing: Digits were normalized to a [0, 1] range, centered, and rescaled.
  • Scale: Models were trained on a subset of 1,000 samples.
  • Fixed Constraints: All models were forced to produce 1024-dimensional codes with exactly 16 active bits to ensure a fair comparison.

Key Insights and Ideas

  • The Sparsity-Information Trade-off: The authors mapped models onto a four-quadrant framework. They found a consistent tension: models that preserve information best (like GAEs with low penalty) often fail to reach maximum sparsity (entropy), while those that reach maximum entropy often lose the data's structure.
  • The GAE Bimodality: Deep learning models (GAEs) showed a "split personality." Depending on how hard they were pushed to be uniform (the KL weight $\lambda$), they either clustered in a high-information/low-entropy zone or a high-entropy/low-information zone, but rarely both.
  • Hebbian Learning Wins on Sparsity: The biologically plausible CSC model was the only one able to reach near-maximal entropy (approaching the theoretical limit of a random code) while still retaining meaningful information.
  • Receptive Field Matters: For the biologically plausible models, where the neurons "look" is critical. Placing receptive fields in a Gaussian distribution centered on the image (CSC-G) significantly outperformed random placement (CSC-R).

Key Results

  • Sparsity: CSC-G reached an entropy ceiling of ~0.99, nearly identical to an optimal random sparse code. In contrast, Dictionary Learning and Random Projections trailed significantly behind.
  • Similarity Preservation: GAEs achieved the highest scores in Class Overlap Separation (COS) and Nearest-Class Impostor Margin (NIM), but only in their low-entropy configurations.
  • kNN Accuracy: CSC-G was the strongest performer here, consistently matching or exceeding the pixel-space baseline. This suggests that its sparse codes actually improved the neighborhood structure of the data compared to the raw input.
  • The "Desired Quadrant": Only CSC-G managed to enter the "High Sparsity / High Information" quadrant, though its information preservation was slightly lower than the best (but less sparse) GAE configurations.

Limitations and Future Work

The primary limitation is the dataset complexity. MNIST is a relatively simple, low-dimensional dataset. The authors acknowledge that it remains an open question whether these findings—particularly the success of biologically plausible Hebbian learning—will generalize to complex, high-dimensional natural images (like ImageNet).

Future work will likely focus on architectural innovations to push models further into the "High Sparsity / High Information" quadrant, potentially by combining local Hebbian learning with more sophisticated structured receptive fields.

Summary updated

Abstract

Biological neural systems achieve impressive computational capabilities under tight metabolic constraints, relying on sparse cortical representations that minimise total neural activity. This combination of efficiency and representational power has motivated growing interest in sparse coding as a computational strategy for artificial systems. Artificial models designed to produce binary sparse codes span three broad paradigms, theoretical methods grounded in efficient coding, deep learning methods trained via backpropagation, and biologically plausible methods that learn through local Hebbian rules, each yielding codes whose quality depends not only on achieving low activation density but also on preserving the informational structure of the input. The latter property, however, lacks an agreed-upon quantitative definition, leaving the field without a principled basis for comparing methods. We propose a set of metrics that formalise information preservation as similarity preservation, measuring how faithfully a binary sparse code reproduces the relational geometry of the input space, and apply this framework to compare four representative methods under matched conditions. The results reveal a consistent trade-off in which models that achieve high information preservation do so at the cost of sparsity, while models approaching the uniformity of an optimal random sparse code tend to sacrifice it. Competitive Sparse Codes, a biologically plausible approach, is the only architecture to occupy a region of near-maximal sparsity while retaining meaningful information preservation. These findings frame the simultaneous satisfaction of both factors as an open challenge for the field and provide a reusable evaluation framework for measuring progress toward it.
BibTeX
@misc{ramos2026binary,
  author = {Jose Miguel Ramos and Luis SaCouto and Andreas Wichert},
  title = {{Binary Sparse Codes: Where are we on Energy Efficient Representations?}},
  year = {2026},
  eprint = {2610.41933},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41933},
}
Figure