Much of modern machine learning operates on real numbers, relying on high-precision floating-point values to represent everything from pixel intensities to physical measurements. However, a significant class of highly efficient hardware platforms (such as FPGAs and optical processors) and unique algorithmic paradigms (such as Weightless Neural Networks and discrete energy-based models) operate natively on bits.
To bridge this gap, we must map continuous scalars into binary strings. Traditional encoding strategies fail this task because they destroy the geometric structure of the original data. Topology-preserving binary encoding—most notably thermometer encoding (or cumulative binary encoding)—solves this problem by ensuring that the geometric relationships of the continuous world are faithfully preserved in the discrete domain.
The Friction Between Continuous Data and Binary Formats
Why can't we just use standard binary numbers or one-hot encodings? Consider mapping a continuous value that ranges from 0 to 10.
If we use standard positional binary encoding (base-2 integers), a tiny change in the continuous value can cause a massive, catastrophic change in the bit pattern. For instance, transitioning from 7 (0111) to 8 (1000) flips every single bit. The Hamming distance between these two adjacent values is maximized, meaning a discrete model looking at these bitstrings would see them as completely unrelated.
If we use one-hot encoding, we divide the range into discrete bins and flip a single bit for the active bin (e.g., bin 2 is 0100, bin 3 is 0010). This treats every bin as completely independent. The distance between bin 2 and bin 3 is exactly the same as the distance between bin 2 and bin 100. One-hot encoding completely erases the concept of scale and order.
Without preserving the topology of the input space, discrete models cannot generalize; they cannot infer that an action taken for an input of 7.0 should be similar to an action taken for 7.1.
The Mechanics of Thermometer Encoding
Thermometer encoding mimics a physical mercury thermometer. As the temperature rises, the column of mercury grows continuously from the bottom up.
To encode a continuous scalar using a thermometer code, we divide the input range into $N$ sequential bins based on incremental thresholds. If a value falls into the $k$-th bin, we set all bits from index 1 up to $k$ to 1, and leave all subsequent bits as 0.
For a 5-bin discretization, the values map as follows:
- Value in Bin 1:
10000 - Value in Bin 2:
11000 - Value in Bin 3:
11100 - Value in Bin 4:
11110
Continuous Value: |----|----|----|----|----|
Bin 2 Representation: 1 1 0 0 0
Bin 3 Representation: 1 1 1 0 0
^ Only 1 bit changes!
This simple mechanism has a profound mathematical property: the Hamming distance between any two encoded vectors is directly proportional to the distance between their original continuous values. Bin 2 (11000) and Bin 3 (11100) differ by exactly 1 bit. Bin 1 (10000) and Bin 4 (11110) differ by 3 bits.
By representing scalar magnitudes as the cumulative count of active bits rather than using positional weights, the encoding preserves the ordinal relationships and topology of the continuous space.
While thermometer encoding is the most common approach, other topology-preserving strategies exist. For instance, Douglas O. Cardoso et al. in Clustering data streams with weightless neural networks and Tobias Schröder et al. in Training Discrete Energy-Based Models with Energy Discrepancy demonstrate that Binary Reflected Gray Codes can similarly preserve proximity as minimal bit-wise differences. However, thermometer encoding remains uniquely powerful because it provides a strong inductive bias, allowing models to allocate distinct parameters to different intensity ranges while naturally respecting their ordering.
Unlocking Weightless Neural Networks
The primary beneficiary of this encoding style is the field of Weightless Neural Networks (WNNs), such as the WiSARD architecture. Unlike traditional deep networks, WNNs do not contain floating-point weights. Instead, they are composed of RAM-based lookup tables. Subsets of the input bitstring are used directly as memory addresses to look up values.
If you pass a positional binary address into a RAM-based neuron, a tiny amount of noise flips bits globally, causing the network to look up a completely random, uninitialized memory address. Generalization becomes impossible.
Thermometer encoding makes WNNs viable for real-world continuous data. As demonstrated by Leopoldo Lusquino Filho et al. in Prediction of palm oil production with an enhanced n-Tuple Regression Network and Alan Tendler Leibel Bacellar et al. in Fast Deep Neural Networks Convergence using a Weightless Neural Model, converting continuous feature vectors into thermometer-encoded strings ensures that similar real-world inputs map to highly overlapping memory addresses. This allows the network to generalize naturally across continuous spaces.
By swapping heavy matrix multiplications for fast bit-addressable lookups, Felipe M.G. França et al. showed in ELM vs. WiSARD: a performance comparison that WNNs can achieve training speeds orders of magnitude faster than floating-point alternatives like Extreme Learning Machines.
This representational strategy is highly versatile across diverse domains:
- Control Systems: Guilherme Souza et al. utilized thermometer encoding in Sun Tracking using a Weightless Q-Learning Neural Network to map continuous environmental variables into stable binary patterns for reinforcement learning.
- Graph Classification: Raul Barbosa et al. in A bag of nodes primer on weightless graph classification applied thermometer encoding to continuous histogram bins, allowing structural graph features to be processed by ultra-fast memory-addressing models.
- Distributed Systems: Otávio O. Napoli et al. leveraged this encoding in Efficient Knowledge Aggregation Methods for Weightless Neural Networks to enable efficient, low-bandwidth federated learning using probabilistic data structures.
Eliminating Floating-Point Bottlenecks in Hardware
In hardware deployments like FPGAs or custom ASICs, floating-point math is incredibly expensive in terms of silicon area, power consumption, and latency. Binary Neural Networks (BNNs) attempt to solve this by restricting weights and activations to $\pm1$ or $0/1$, turning expensive multiplications into cheap bitwise XNOR operations.
However, traditional BNNs hit a bottleneck at the very first layer. Because real-world inputs (like camera pixels) are continuous, the first layer usually must remain floating-point to avoid destroying the input signal's fidelity.
Thermometer encoding completely removes this floating-point bottleneck. In FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations, Junhao Pan et al. binarized the input layer of a deep network using thermometer encoding. Because feature similarity is preserved in the bitstream, the very first layer of the network can execute using pure binary multiply-accumulate operations, enabling a completely hardware-efficient, end-to-end binary pipeline without accuracy degradation.
A parallel breakthrough exists in analog computing. In Optical Reservoir Computing Using Multiple Light Scattering for Chaotic Systems Prediction, Jonathan Dong et al. mapped continuous inputs into multi-pixel binary patterns (termed "basket encoding") to interface with optical hardware. This topology-preserving mapping ensured that similar inputs yielded stable, predictable physical states within an optical scattering medium, allowing low-precision optical hardware to match the performance of high-precision electronics.
Breaking Linear Extrapolation for Adversarial Robustness
Beyond hardware efficiency, topology-preserving binarization alters how deep neural networks process information. Standard deep networks rely on continuous, piecewise-linear activation functions (like ReLU). This creates a security flaw: an attacker can calculate a tiny, coordinated perturbation along the network's gradient. Because the network behaves linearly over large ranges, this tiny input change extrapolates into a massive change in the final output layers, causing a catastrophic misclassification (an adversarial attack).
In Thermometer Encoding: One Hot Way To Resist Adversarial Examples, Jacob Buckman et al. demonstrated that preprocessing continuous pixels into thermometer-encoded vectors drastically improves adversarial robustness.
By forcing the network to operate on a discrete, binarized representation of the input space, the smooth, continuous relationship between the raw input and the internal activations is broken. An attacker can no longer compute a reliable gradient to exploit via linear extrapolation. Crucially, because the encoding is order-preserving, the network retains its ability to learn accurate, high-performing decision boundaries, gaining security without sacrificing standard performance.
Key Foundational Work
Clustering data streams with weightless neural networks (2011) — contributed 2 concept elements to this direction.
Comparing and Combining Generative and Posterior Probability Models: Some Advances in Sentence Boundary Detection in Speech (2004) — contributed 1 concept elements to this direction.