Deep neural networks excel at processing continuous representations, but real-world deployment often demands discrete, binary outputs. Whether compressing models for edge devices, building lightning-fast image retrieval systems via hashing, or generating low-poly 3D meshes, representing data as binary bits (such as 0 and 1, or -1 and +1) dramatically reduces memory footprints and accelerates computation.
However, forcing a network to output binary values introduces a fundamental mathematical conflict. The standard mathematical operation to binarize a value is the step or sign function:
$$ \text{sign}(x) = \begin{cases} +1 & \text{if } x \geq 0 \\ -1 & \text{if } x < 0 \end{cases} $$
The derivative of this function is zero everywhere and undefined at zero. If a neural network includes a hard binarization step in its forward pass, the gradient during backpropagation becomes zero, completely halting learning. This is known as the vanishing gradient problem for discrete representations.
To bridge this gap, researchers use differentiable binary approximations—replacing the non-differentiable step function with smooth, continuous relaxations during training, and gradually tightening them into hard binary constraints.
The Core Mechanism: Smooth Proxies
The most intuitive way to make binarization differentiable is to replace the step function with a smooth, S-shaped curve that mimics its behavior but retains non-zero gradients.
For instance, in unsupervised image matching, Maciej Zieba et al. utilized a soft-sign proxy function in BinGAN: Learning Compact Binary Descriptors with a Regularized GAN to allow gradients to flow through a compact penultimate layer. Similarly, for open-set category discovery in LiDAR segmentation, Hyeonseong Kim et al. adopted a hyperbolic tangent ($\tanh$) function as a differentiable proxy for the sign operation in On-the-Fly Category Discovery for LiDAR Semantic Segmentation.
Other variations adapt the proxy function to the target output range. When the goal is to output bits in the range $[0, 1]$ rather than $[-1, 1]$, a parameterized sigmoid function is often preferred. In optical wireless communications, Hoon Lee et al. used a sigmoid function to approximate binary switches in Deep Learning Framework for Wireless Systems: Applications to Optical Wireless Communications. In image compression, Wen-Hsiao Peng et al. represented discrete bits as a differentiable superposition of sigmoid functions in Learned Image Compression with Soft Bit-based Rate-Distortion Optimization, enabling the network to optimize both image distortion and bit-rate simultaneously.
The Continuation Method: Annealing the Relaxation
While a static smooth proxy allows gradients to flow, it introduces a new problem: quantization error. If a network trains entirely on a smooth $\tanh$ function, the final outputs will be continuous values (e.g., $0.75$ or $-0.3$) rather than strict binary values ($+1$ or $-1$). Forcing these values to their nearest binary extremes during deployment causes a severe drop in performance.
To solve this, researchers employ a continuation method (or deterministic annealing). The idea is to start training with a very smooth approximation to ensure stable gradient flow, and progressively scale the steepness of the function over time until it converges into a sharp, non-differentiable step function.
Mathematically, this is often achieved by introducing a scaling parameter $\beta$ into the activation function, such as $\tanh(\beta x)$. As $\beta \to \infty$, $\tanh(\beta x)$ converges exactly to $\text{sign}(x)$.
In his work on deep hashing, Zhangjie Cao demonstrated this approach in HashNet: Deep Learning to Hash by Continuation, showing that slowly increasing $\beta$ minimizes quantization error by easing the network into discrete constraints. This continuation strategy has been widely adopted across domains:
- Cross-Modal Retrieval: Shupeng Su et al. applied annealed activation sharpening to bridge the modality gap between text and images in Deep Joint-Semantics Reconstructing Hashing for Large-Scale Unsupervised Cross-Modal Retrieval.
- Information Retrieval: Ikuya Yamada et al. used a scaled $\tanh$ continuation strategy in Efficient Passage Retrieval with Hashing for Open-domain Question Answering to compress massive passage embeddings down to compact bitstrings.
- Hardware Optimization: The authors of Training Progressively Binarizing Deep Networks Using FPGAs leveraged a continuation approach to gradually transition fixed-point parameters into binary states directly on hardware, avoiding the instability of sudden binarization.
Beyond Activations: Differentiable Quantization Losses
Instead of changing the activation function, another powerful way to enforce binarization is through differentiable loss functions that penalize non-binary states. By designing a smooth, U-shaped loss function, the optimization landscape naturally pushes real-valued weights or activations toward the desired binary extremes.
For example, Feng Zheng 0001 et al. introduced smooth quantization losses based on "log-cosh" and binary entropy in Binarized Neural Networks for Resource-Efficient Hashing with Minimizing Quantization Loss. The log-cosh loss acts as a differentiable surrogate that penalizes parameters deviating from $+1$ or $-1$, providing smooth gradients that guide the network toward discrete states without blocking backpropagation.
Similarly, Han Zhu 0007 et al. modeled binary constraints as a probabilistic prior within a Bayesian framework in Deep Hashing Network for Efficient Similarity Retrieval, effectively nudging continuous outputs toward discrete values while maintaining end-to-end differentiability.
Decoupling the Forward and Backward Passes
A more radical approach to binarization involves using different functions for the forward and backward passes. In the forward pass, the network uses the true, hard discrete function to ensure the model behaves exactly as it would during deployment. In the backward pass, it substitutes a differentiable surrogate function to compute gradients.
This concept underpins the Straight-Through Estimator (STE) and its modern theoretical extensions. For instance, Daniel Mas Montserrat et al. implemented a differentiable binarization layer that uses a hard threshold for forward passes and a continuous approximation for backpropagation in Generative Moment Matching Networks for Genotype Simulation.
To provide formal convergence guarantees for these types of split-pass networks, Yiwei Lu et al. proposed the ProxConnect++ framework in Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers. They showed that by carefully pairing specific forward activations (like sign-Swish) with their mathematical derivatives in the backward pass, one can formally optimize smoothed surrogates of the binary objective function while maintaining mathematical tractability.
Diverse Applications of Differentiable Binarization
The ability to optimize discrete representations via continuous relaxations has unlocked solutions across a wide array of machine learning challenges:
- Object Detection and Segmentation: In Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion, Minghui Liao et al. incorporated a differentiable binarization step that allows a neural network to jointly optimize a probability map and a learned threshold map, enabling the model to adaptively set sharp boundaries for text detection.
- 3D Geometry: Zhiqin Chen et al. developed BSP-Net: Generating Compact Meshes via Binary Space Partitioning, which uses a continuous relaxation of binary logic during training to output clean, structured, watertight polygonal meshes.
- Discrete Representation Learning: In LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes, Aditya Kusupati et al. bypassed the non-differentiable sign function using a gradient approximation technique, allowing networks to learn extremely compact 20-bit representations for ImageNet classification and retrieval.
By transforming non-differentiable step functions into smooth, trackable landscapes, differentiable binary approximations allow deep learning models to retain the power of gradient descent while achieving the extreme efficiency of discrete representations.
Key Foundational Work
kNN Hashing with Factorized Neighborhood Representation (2015) — contributed 2 concept elements to this direction.