Optimization Strategies for Discrete Weight Neural Networks

This direction is merged into another direction: discrete weight optimization. Its last version stays readable and citable here.

Version 1 of 171 papers75 conceptsfrozen 2026-10-04

Deep learning is fundamentally built on the assumption of continuity. We use backpropagation to nudge high-precision, floating-point weights along a smooth loss landscape. However, deploying these models on edge devices—smartwatches, sensors, or satellites—demands a radical departure from this paradigm. To save power and memory, we often need discrete weight networks, where synapses are restricted to binary ($\{-1, 1\}$) or ternary ($\{-1, 0, 1\}$) values.

The challenge is that discrete spaces are "jagged." As Daniel Soudry and Ron Meir noted in Mean Field Bayes Backpropagation, the step functions required for quantization have gradients that are zero almost everywhere and undefined at the jumps. This makes training an NP-hard combinatorial problem, rendering standard stochastic gradient descent (SGD) ineffective.

To bridge this gap, researchers have developed four primary strategies to optimize neural networks without relying on traditional continuous derivatives.

The Bayesian Bridge: Learning Distributions, Not Values

One way to handle discrete weights is to stop treating them as fixed values during training. Instead, we can treat them as random variables. By learning the parameters of a probability distribution (like the mean and variance of a Gaussian) that represents the weights, we restore a smooth landscape for optimization.

In Training Binary Neural Networks using the Bayesian Learning Rule, the authors demonstrate that by optimizing the distribution rather than the discrete weights themselves, we can derive principled updates that resemble popular heuristics like the Straight-Through Estimator (STE) but with stronger theoretical grounding. Similarly, VISPA by Lorenzo Orecchia and colleagues uses Gaussian variational inference to model correlations between weights, turning a discrete search into a continuous optimization of a variational posterior.

Decoupling Constraints via ADMM

A second approach treats quantization as a constrained optimization problem: "Minimize the loss function, subject to the constraint that every weight must belong to a discrete set."

The Alternating Direction Method of Multipliers (ADMM) is a powerful tool for this. As shown by Cong Leng in Extremely Low Bit Neural Network, ADMM allows us to decouple the objective. We alternate between two steps:

  1. The Loss Step: Update the weights to minimize the task error (ignoring the discrete constraint).
  2. The Projection Step: Nudge the weights toward the nearest valid discrete values (binary or ternary).

This "tug-of-war" between the task objective and the hardware constraint allows models to converge on low-bit representations that maintain high accuracy. Shaokai Ye and Yanzhi Wang extended this in their Unified Framework, achieving massive compression ratios by combining pruning and quantization within the same ADMM loop.

Optimal Control and Non-Gradient Dynamics

If gradients are the problem, why use them at all? Some researchers have looked toward Optimal Control Theory to find a new mathematical foundation for deep learning.

In An Optimal Control Approach to Deep Learning, Qianxiao Li and Shuji Hao reformulate the layers of a network as time-steps in a dynamical system. By applying Pontryagin’s Maximum Principle (PMP), they developed the Method of Successive Approximations (MSA). Unlike backpropagation, which flows gradients backward, MSA solves for the optimal discrete weights by maximizing a Hamiltonian function at each layer. This provides a mathematically rigorous way to train binary and ternary networks where gradients are otherwise undefined.

Discrete Inference and Message Passing

Finally, we can view a neural network not as a function to be differentiated, but as a factor graph—a probabilistic model where weights are variables and training data are constraints.

Carlo Baldassi and Alfredo Braunstein have pioneered this view in works like A Max-Sum algorithm for training discrete neural networks. Instead of descending a gradient, they use message-passing algorithms (like Belief Propagation) to explore the discrete weight space globally. This approach is particularly effective at finding "dense" regions of the weight space—clusters where many valid solutions exist—which Baldassi argues are more robust and generalizable than the isolated points found by local search.

Why It Works: The Geometry of Discreteness

The recurring theme across these papers is that discreteness acts as a regularizer. While it makes optimization harder, it also prevents the model from over-fitting to high-frequency noise. Kaiqi Zhang and Yu-Xiang Wang proved in Why Quantization Improves Generalization that binary weight networks behave like "quasi-neural networks" with smoother kernels, effectively forcing the model to focus on the most important features.

By moving beyond simple rounding heuristics and embracing these sophisticated optimization frameworks, the field is proving that we can have the best of both worlds: the high performance of deep learning and the extreme efficiency of discrete hardware.

Papers in this version

Showing 60 of 71 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimization_strategies_for_discrete_wei_8881fdb6,
  title = {Optimization Strategies for Discrete Weight Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimization-strategies-for-discrete-weight-neural-networks/rdr_8881fdb65b50453aa69e8dfb6b4c5fba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimization_strategies_for_discrete_wei_8881fdb6_v1,
  title = {Optimization Strategies for Discrete Weight Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimization-strategies-for-discrete-weight-neural-networks/rdr_8881fdb65b50453aa69e8dfb6b4c5fba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}