Low-precision architectures, such as Binarized Neural Networks (BNNs), offer massive efficiency gains during inference. By constraining weights and activations to just 1 bit ($\pm1$), they replace power-hungry floating-point multiplications with simple bitwise XNOR and popcount operations.
However, a deep engineering paradox emerges when we try to train these networks. While the final model fits comfortably on resource-constrained edge devices, the training phase remains tethered to high-performance computing environments. This training-inference memory asymmetry exists because standard optimization algorithms rely on high-precision auxiliary variables—traditionally called latent weights, master weights, or shadow weights—to accumulate gradient updates.
A growing body of research aims to break this high-precision dependency, allowing neural networks to be optimized with the same bit-level efficiencies that characterize their inference.
The Core Problem: Quantization Underflow
To understand why latent weights were introduced in the first place, we have to look at how standard gradient descent interacts with discrete values.
Imagine a single weight $w$ in a BNN that is currently binarized to $+1$. During backpropagation, the optimizer computes a small gradient update based on continuous calculus. Let's say the update step is:
$$\Delta w = -\eta \cdot g = -0.0005$$
If we try to apply this update directly to our binary weight, we run into an immediate mathematical wall:
$$w_{\text{new}} = \text{sign}(1 - 0.0005) = \text{sign}(0.9995) = +1$$
Because the update is too small to overcome the discretization threshold, the weight does not change. The gradient update is completely lost to rounding error—a phenomenon known as quantization underflow. If this happens across the entire network, the model cannot learn.
To circumvent this, early foundational works like Binarized Neural Networks by Itay Hubara et al. and XNOR-Net by Mohammad Rastegari et al. maintained a hidden, full-precision (FP32) copy of the weights. Small gradient updates accumulate in this latent copy over many iterations. When the latent weight finally crosses zero, the binarized weight used in the forward pass flips from $+1$ to $-1$ (or vice versa).
Standard BNN Training Loop:
[FP32 Latent Weight] ---> Accumulates tiny gradients (-0.0005, -0.0005...)
|
v (Binarization / Sign Function)
[ 1-Bit Weight ] ---> Used for efficient Forward/Backward passes
As highlighted by Tao Luo et al. in Adaptive Precision Training for Resource Constrained Devices, this creates a severe memory bottleneck. Even if your forward pass uses 1-bit math, your optimizer memory footprint is dominated by 32-bit floating-point master weights and gradient buffers. This overhead prevents the deployment of full training pipelines on edge hardware or resource-constrained clients, such as those in decentralized split learning setups (Ngoc Duy Pham et al. in Binarizing Split Learning for Data Privacy Enhancement and Computation Reduction).
Rethinking Latent Weights as Inertia
To eliminate latent weights, we must first understand what role they actually play during optimization. Koen Helwegen et al. provided a vital conceptual shift in Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization. They argued that real-valued latent weights are not actually "weights" because their absolute magnitudes do not affect the forward pass; only their sign matters.
Instead, a latent weight acts as a state of inertia. It records how consistently gradients have been pushing a weight in a given direction. If a latent weight is deeply positive, it means many consecutive negative gradients are required to force a bit-flip.
Recognizing this, the authors introduced the Binary Optimizer (Bop). Instead of tracking a continuous floating-point position, Bop tracks inertia explicitly. A bit flips only when the gradient maintains a consistent sign across a specific time horizon.
Following this trajectory, Nianhui Guo et al. introduced Diode in Enhancing Optimization Robustness in 1-Bit Neural Networks Through Stochastic Sign Descent. Diode completely abandons latent weights by utilizing a stochastic sign descent approach. Rather than tracking noisy gradient magnitudes in high precision, it focuses entirely on the direction (sign) of the gradients, updating binary states stochastically to cut memory usage by up to 50%.
Native Discrete and Boolean Optimization
If continuous calculus forces us to use high-precision tracking, why not abandon calculus altogether? A parallel branch of research focuses on training networks natively within discrete or Boolean domains.
In BOLD: Boolean Logic Deep Learning, Van Minh Nguyen et al. introduced a mathematical framework called "Boolean variation." Instead of calculating continuous derivatives, BOLD evaluates how discrete changes in Boolean states propagate through the network, allowing native training without real-valued gradients or latent weights.
For structured embeddings, a similar philosophy applies. Katsuhiko Hayashi 0001 et al. developed a greedy bit-flip training algorithm in A Greedy Bit-flip Training Algorithm for Binarized Knowledge Graph Embeddings. Rather than computing floating-point updates and passing them through a straight-through estimator, their approach evaluates local discrete changes directly, flipping bits based on a greedy combinatorial strategy that completely bypasses shadow vectors.
In fact, Han Zhou formalized this perspective in Combinatorial optimization for low bit-width neural networks by framing risk minimization as a submodular optimization problem. By treating network optimization as a discrete search rather than a continuous descent, the author demonstrated that models can achieve global approximation guarantees using purely low-cost bit operations.
Multi-Component and Error-Tracking Representations
For massive models like Large Language Models (LLMs), completely abandoning gradient-based optimization can be difficult due to convergence stability. For these architectures, researchers mitigate the latent weight overhead by using multi-component or multi-kernel representations that track precision errors without requiring a full FP32 master copy.
Ba-Hien Tran developed the Multiple Boolean Kernels (MBOK) framework in Highly Efficient and Effective LLMs with Multi-Boolean Architectures. Instead of keeping a single high-precision weight matrix that gets quantized down, MBOK represents weights as a linear combination of multiple discrete Boolean bases. Fine-tuning is performed directly on these Boolean kernels, eliminating the memory-heavy latent copies while matching the accuracy of traditional multi-bit quantization methods.
Traditional Quantization-Aware Training:
[ FP32 Master Weight Matrix ] <-- Massive Memory Overhead
MBOK Framework:
[ Boolean Kernel 1 ] + [ Boolean Kernel 2 ] + [ Boolean Kernel 3 ] <-- Optimized Directly
For mixed-precision training where weights are kept in formats like BF16, Tao Yu et al. designed Collage: Light-Weight Low-Precision Strategy for LLM Training. Instead of maintaining a full 32-bit master copy to prevent rounding errors, Collage utilizes a multi-component floating-point representation at critical steps. This allows the training pipeline to run entirely in low-precision formats, tracking rounding errors dynamically and achieving significant memory savings without sacrificing convergence stability.
By shifting from storing high-precision states to tracking discrete inertia, Boolean variations, or multi-component errors, this research direction successfully resolves the training-inference memory paradox, bringing edge-native training closer to reality.
Key Foundational Work
Binarized Neural Networks (2016) — contributed 1 concept elements to this direction.
Towards Accurate Binary Convolutional Neural Network (2017) — contributed 1 concept elements to this direction.
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2017) — contributed 1 concept elements to this direction.