Spike and Slab Priors for Sparse Neural Networks

25 concepts22 papers1 proposal

Deep neural networks are notoriously overparameterized. To make them efficient, we often want to prune away unnecessary weights. Traditional methods rely on optimization heuristics, such as training a dense network and then zeroing out weights below a certain threshold, or applying continuous regularization like $L_1$ (Lasso) penalties.

However, continuous penalties merely shrink weight magnitudes rather than setting them exactly to zero. To achieve true parameter elimination within a rigorous probabilistic framework, researchers turn to the spike-and-slab prior. This approach treats network architecture selection not as a post-processing trick, but as a core part of Bayesian inference.

The Intuition behind the Mixture Model

The core idea of a spike-and-slab prior is to model each weight $w$ as a draw from a mixture of two distinct distributions:

$$p(w) = (1 - \pi)\cdot\text{Spike}(w) + \pi\cdot\text{Slab}(w)$$

The spike component is highly concentrated around zero. In its most rigid form, it is a Dirac delta function $\delta_0(w)$, which places a point mass exactly at zero. If a weight is drawn from the spike, it is completely inactive—effectively pruned from the network.

The slab component is a broad, continuous distribution (such as a wide Gaussian, Laplace, or Uniform distribution) that allows the weight to take on a wide range of values if it is deemed important. The parameter $\pi$ acts as a prior probability of a weight being active.

As Veronika Rocková et al. demonstrate in Posterior Concentration for Sparse Deep Learning, this formulation functions as a Bayesian variant of $L_0$ regularization. Instead of softly biasing weights to be small, it forces a stark, discrete choice: a connection is either completely absent or fully functional. Jincheng Bai et al. emphasize in Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee that utilizing a true Dirac delta function as the spike enables exact parameter elimination during the forward pass, distinguishing it from continuous shrinkage priors that require arbitrary post-hoc thresholding.

Continuous Prior (e.g., Laplace)       Spike-and-Slab Prior
          ▲                                      ▲
          │       |                              │       █  <- Dirac Spike
          │      / \                             │       █
          │     /   \                            │     / █ \
          │   _/     \_                          │   _/  █  \_  <- Continuous Slab
   ───────┴─────────────┴──────►          ───────┴─────────────┴──────►
                w = 0                                  w = 0

Overcoming the Optimization Challenge

If spike-and-slab priors are so theoretically appealing, why aren't they used everywhere? The challenge lies in the non-differentiable nature of the Dirac delta function. Standard backpropagation relies on smooth gradients, but a point mass at zero introduces a discrete discontinuity.

To bridge this gap, researchers employ several clever relaxation and sampling strategies:

1. Latent Binary Gating

Instead of modifying the weight distribution directly, we can introduce a latent binary variable $z \in \{0, 1\}$ for each weight, acting as a stochastic gate. In Sparsifying Bayesian neural networks with latent binary variables and normalizing flows, Lars Skaaret-Lund et al. use these binary variables to determine whether a weight is active ($z=1$) or pruned ($z=0$). To make this trainable, frameworks often use continuous relaxations like the Gumbel-Softmax trick, allowing gradients to flow through the discrete selection mechanism.

2. Continuous Approximations of the Spike

Another approach replaces the infinitely thin Dirac delta with a continuous, but highly concentrated, distribution. For instance, Guang Lin et al. develop An Adaptive Empirical Bayesian Method for Sparse Deep Learning, which pairs a narrow Laplace distribution (the spike) with a wider Normal distribution (the slab). Similarly, Karen Ullrich et al. implement a Gaussian Mixture Model prior in Soft Weight-Sharing for Neural Network Compression, fixing one component at zero with high initial probability to create a default state of sparsity. Weights must significantly improve the model's objective to escape this zero-cluster.

3. Reformulating Existing Heuristics

Remarkably, some of deep learning's most common heuristics can be reframed as spike-and-slab inference. In Dropout as a Structured Shrinkage Prior, Eric Nalisnick et al. prove that multiplicative noise, such as Bernoulli dropout, is mathematically equivalent to a spike-and-slab prior. This reveals that dropout is not just an empirical trick to prevent overfitting, but a formal variable selection mechanism operating under the hood.

Why the Bayesian Approach Works

Beyond compression, treating sparsity probabilistically yields significant theoretical and practical advantages over deterministic pruning.

Theoretical Guarantees

Approximating exact posteriors in deep networks is notoriously difficult, but the spike-and-slab framework offers strong mathematical safety nets. Badr-Eddine Chérief-Abdellatif proves in Convergence Rates of Variational Inference in Sparse Deep Learning that variational inference approximations for these sparse networks achieve optimal convergence rates, meaning the approximation is as mathematically sound as exact Bayesian inference. Furthermore, Keitaro Sakamoto et al. utilize this prior within a PAC-Bayesian framework in Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective to derive generalization bounds that account for both network sparsity and weight magnitude simultaneously.

Robustness and Calibration

Deterministic sparse networks often suffer from miscalibration—becoming overconfident in their predictions. By maintaining a posterior distribution over the weights, spike-and-slab networks preserve reliable uncertainty estimates. Yan Sun et al. show in Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration that this probabilistic representation prevents the optimization process from getting stuck in poor local minima. Additionally, Ozan Ozdenizci et al. demonstrate in Training Adversarially Robust Sparse Networks via Bayesian Connectivity Sampling that estimating the posterior of network connectivity from the very first iteration yields models that are inherently resilient to adversarial attacks without needing a dense pre-training phase.

Applications Beyond Standard Deep Learning

The ability to cleanly separate "signal" from "noise" makes spike-and-slab priors highly effective in domains where data is sparse, high-dimensional, or requires strict interpretability.

By formalizing the distinction between zero and non-zero weights, the spike-and-slab prior provides a mathematically rigorous foundation for network pruning, transforming model compression from an empirical art into a principled inference process.

Key Foundational Work

Soft Weight-Sharing for Neural Network Compression (2017) — contributed 1 concept elements to this direction.
Model-based Bayesian inference of neural activity and connectivity from all-optical interrogation of a neural circuit (2017) — contributed 1 concept elements to this direction.
Bayesian Inference and Online Experimental Design for Mapping Neural Microcircuits (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spike_and_slab_prior_probabilistic_varia_b307f62e,
  title = {spike-and-slab prior (probabilistic variable selection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spike-and-slab-prior-probabilistic-variable-selection/rdr_b307f62e757444a4af71718ef93aa421},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_spike_and_slab_prior_probabilistic_varia_b307f62e_v1,
  title = {spike-and-slab prior (probabilistic variable selection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spike-and-slab-prior-probabilistic-variable-selection/rdr_b307f62e757444a4af71718ef93aa421/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake