Approximating High Precision Weights with Multiple Binary Bases

40 concepts31 papers1 proposal

Standard Binary Neural Networks (BNNs) offer an alluring promise: compress a deep learning model by forcing its weights and activations into a single bit ($\pm1$). In a standard BNN, such as the seminal XNOR-Net framework introduced by Mohammad Rastegari et al. in [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks], a high-precision weight matrix $W$ is approximated using a single real-valued scalar $\alpha$ and a binary matrix $B$:

$$W \approx \alpha B$$

This representation is incredibly hardware-friendly. It replaces power-hungry floating-point multiplications with efficient bitwise XNOR and bit-count operations. However, this extreme quantization introduces a severe expressivity crisis. Collapsing a continuous space of real numbers into just two states per element causes massive quantization errors, leading to catastrophic accuracy drops on complex, large-scale datasets.

To bridge this performance gap without sacrificing the speed of bitwise logic, a powerful paradigm has emerged: multi-basis binary decomposition. Instead of forcing a hard choice between ultra-low-fidelity 1-bit representations and expensive 32-bit floating-point numbers, this approach decomposes high-precision tensors into a linear combination or product of multiple binary bases.

The Mechanics of Additive Binary Decomposition

The core intuition behind additive binary decomposition is simple: if one binary matrix cannot accurately represent a continuous weight distribution, an ensemble of them can. Mathematically, the weight matrix is expanded into a weighted sum of $M$ independent binary bases:

$$W \approx \sum_{i=1}^M \alpha_i B_i$$

where each $B_i \in \{-1, +1\}^{m \times n}$ is a binary matrix, and each $\alpha_i$ is a real-valued scaling factor.

By adjusting the number of bases $M$, researchers can smoothly trade off accuracy and computational overhead. When $M=1$, the system reverts to a standard BNN. As $M$ increases, the approximation error drops exponentially, allowing the network to reconstruct the original continuous weight distribution with high fidelity.

Xiaofan Lin et al. demonstrated the potency of this strategy with ABC-Net in [Towards Accurate Binary Convolutional Neural Network], proving that multi-base binarization dramatically narrows the performance gap on challenging benchmarks like ImageNet. Rather than training these bases entirely from scratch, the decomposition can also be applied directly to pre-trained, high-precision models. For instance, Steven Y. Guo et al. introduced Network Sketching in [Network Sketching: Exploiting Binary Structure in Deep CNNs], a method that decomposes complex convolutional filters into a weighted sum of binary bases at deployment time, offering a granular control over the model's memory footprint.

This structural flexibility means that compression can happen without a lengthy retraining loop. Ryuji Kamiya et al. showed in [Binary-Decomposed DCNN for Accelerating Computation and Compressing Model Without Retraining] that decomposing pre-trained weights into a binary basis matrix and a scaling vector preserves structural information well enough to achieve immediate size reductions with minimal accuracy loss. Furthermore, Kun Huang demonstrated in [Efficient Quantization for Neural Networks with Binary Weights and Low Bitwidth Activations] that this multi-level binarization strategy preserves the expressivity required for large-scale visual recognition while shifting the bulk of the arithmetic workload to low-cost operations.

Beyond Simple Sums: Matrix Products and Higher-Order Expressivity

While additive combinations are highly effective, they are not the only way to decompose a matrix. A parallel line of research leverages the multiplicative properties of binary matrices to achieve a combinatorial increase in numerical diversity.

Instead of summing independent matrices, the Semi-Binary Decomposition (SBD) framework proposed by Qinghao Hu et al. in [Training Binary Weight Networks via Semi-Binary Decomposition] factorizes a high-precision weight matrix into the product of two binary matrices bounded by a diagonal scaling matrix:

$$W \approx B_1 D B_2$$

This multiplicative structure allows a network to generate a vast array of possible weight magnitudes from just a few binary components. Because the product of two binary values ($\pm1$) remains binary, the underlying operations remain highly efficient, yet the resulting matrix captures a far richer spectrum of continuous values.

This concept has recently been adapted to compress Large Language Models (LLMs). Vladimír Boža et al. introduced Double Binary Factorization in [Addition is almost all you need: Compressing neural networks with double binary factorization], demonstrating that this product-based binary representation maintains lower perplexity at extreme compression levels (1-bit and 2-bit ranges) compared to traditional scalar quantization techniques.

To push expressivity even further, Kyo Kuroki proposed Binary Quadratic Quantization (BQQ) in [Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression]. BQQ moves beyond first-order linear combinations by utilizing second-order expressions—sums of binary matrix products. This mathematical formulation captures intricate row-column correlations and effectively acts as a non-linear feature generator, allowing ultra-low bit-width models to perform accurately even in data-free post-training scenarios.

Why It Works: Hardware Acceleration via Bit-Serial Math

The true genius of multi-basis binary decomposition lies in how it executes on actual hardware. To compute a standard matrix multiplication $Y = WX$ using an additively decomposed weight matrix, we can distribute the activation matrix $X$ across the bases:

$$Y = \left( \sum_{i=1}^M \alpha_i B_i \right) X = \sum_{i=1}^M \alpha_i (B_i X)$$

Because every element in $B_i$ is either $+1$ or $-1$, the intermediate multiplication $B_i X$ requires absolutely no floating-point multiplications. Hardware can compute $B_i X$ entirely through simple additions and subtractions (or sign-flips). The real-valued scaling factors $\alpha_i$ are only applied at the very end of the pipeline.

High-Precision Weight Matrix (W)
       │
       ▼ [Decomposition]
 ┌───────────┬───────────┬───────────┐
 │  Alpha_1  │  Alpha_2  │  Alpha_M  │  (Real-Valued Scaling Vectors)
 ├───────────┼───────────┼───────────┤
 │  Basis_1  │  Basis_2  │  Basis_M  │  (Binary Sign Matrices: ±1)
 └─────┬─────┴─────┬─────┴─────┬─────┘
       │           │           │
       ▼           ▼           ▼  [Parallel Bitwise / Additive Ops]
   (B_1 × X)   (B_2 × X)   (B_M × X)    (No floating-point multiplications)
       │           │           │
       └───────────┼───────────┘
                   ▼ [Weighted Summation]
             Final Output (Y)

This bit-serial decomposition ensures that the computational cost scales linearly with the bit-width of the weights, avoiding the performance plateaus that plague traditional mixed-precision hardware kernels. To maximize this efficiency, the authors of [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge] implemented matrix multiplication using bit-serial decomposition paired with fast table lookups. This design completely bypasses the traditional "bit-unpacking" overhead that usually slows down low-precision models on standard CPUs.

Similarly, Yongkweon Jeon et al. developed BiQGEMM in [BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs], which uses lookup tables to replace redundant arithmetic operations entirely, accelerating binary-coded quantized networks on standard, consumer-grade hardware.

Scaling to Modern Transformers and Activations

As the deep learning landscape has shifted from convolutional networks to massive Transformer architectures, multi-basis binary decomposition has proven essential for deploying Large Language Models (LLMs) and Vision Transformers (ViTs) to resource-constrained edge devices.

Standard 1-bit quantization fails completely on LLMs due to the presence of high-magnitude emergent outliers in the weight matrices. To combat this, Wanxiang Che et al. introduced OneBit in [OneBit: Towards Extremely Low-bit Large Language Models], which pairs a binary sign matrix with two low-rank, high-precision floating-point scaling vectors. This hybrid architecture ensures that the vast majority of operations occur via bitwise math, while the low-rank vectors stabilize the model by preserving outlier dimensions.

For scenarios requiring dynamic performance tuning, Gunho Park et al. proposed AnyBCQ in [AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs]. AnyBCQ allows a single pre-trained LLM to alter its bit-width (operating at 2, 3, or 4 bits) on the fly. It achieves this by progressively adding or removing binary bit-planes during inference, avoiding the hardware bottlenecks and centroid lookup tables required by non-uniform quantization methods.

A similar challenge exists in Vision Transformers, where sensitive components like attention matrices are highly prone to quantization degradation. Tian Gao et al. addressed this in [GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples] by using a linear combination of multiple binary masks to safeguard critical attention structures.

Finally, this decomposition principle is not restricted to weights; it applies equally to activation tensors. Gang Hua et al. in [How to Train a Compact Binary Neural Network with High Accuracy?] and Fabrizio Pedersoli et al. in [Espresso: Efficient Forward Propagation for Binary Deep Neural Networks] demonstrated that multi-bit activation inputs can be sliced into separate binary bit-planes. These planes are processed independently through the network's binary layers and subsequently recombined via a final weighted sum. This ensures that the entire computation pipeline remains end-to-end bitwise, delivering massive hardware speedups without sacrificing structural fidelity.

Key Foundational Work

Towards Accurate Binary Convolutional Neural Network (2017) — contributed 2 concept elements to this direction.
Network Sketching: Exploiting Binary Structure in Deep CNNs (2017) — contributed 2 concept elements to this direction.
Binary-Decomposed DCNN for Accelerating Computation and Compressing Model Without Retraining (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_binary_weight_decomposition_b8b57fb5,
  title = {binary weight decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-weight-decomposition/rdr_b8b57fb5b9b64e02811b0df58c85ed91},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_binary_weight_decomposition_b8b57fb5_v1,
  title = {binary weight decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-weight-decomposition/rdr_b8b57fb5b9b64e02811b0df58c85ed91/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake