Beyond a Single Sign: BiTE for Branch-Preserving Binary Threshold Expansion

Siddharth Gupta, Mikail Yayla, Akash KumararXiv 2026 · arXiv

TL;DR

BiTE (Branch-Preserving Binary Threshold Expansion) boosts the performance of Binary Neural Networks (BNNs) by replacing the standard single-threshold binarization with multiple learnable thresholds. By keeping these multiple binary responses separate rather than collapsing them into a single value, BiTE allows the network to learn more complex, state-specific transformations, achieving higher accuracy with fewer parameters.

What Problem Does This Paper Address?

Binary Neural Networks are the "lightweight champions" of deep learning, using 1-bit weights and activations to drastically reduce memory and computation costs. However, this efficiency comes at a steep price: representational loss.

Standard BNNs use a simple sign function to binarize data—anything above zero becomes +1, and anything below becomes -1. This "single sign" approach throws away all information about magnitude and variation within those regions. While previous researchers have tried using multiple thresholds to create more activation states, they typically "collapse" these states back into a single scalar value before passing them to the next layer. The authors argue that this "scalar collapse" restricts the network's ability to learn, as it forces the next layer to treat different threshold crossings as having the same linear effect.

Methods

The authors introduce BiTE, which stands for Branch-Preserving Binary Threshold Expansion. The core idea is two-fold:

  1. Multiple Ordered Thresholds: Instead of one threshold at zero, BiTE learns $K$ ordered thresholds for each feature direction. This allows the network to distinguish between $K+1$ different activation states.
  2. Branch Preservation: This is the "secret sauce." Instead of averaging or summing these $K$ binary responses into one number, BiTE keeps them as separate, independent inputs for the next layer. This allows the subsequent operator to assign unique weights to each specific threshold crossing.
Comparison of standard activation, branch-collapsed expansion, and the proposed branch-preserving BiTE.

The authors implemented BiTE in two ways:

  • Fixed-width Convolutional layers: They trade off the number of independent feature directions for more thresholds per direction. For example, instead of 64 directions with 1 threshold each, they might use 32 directions with 2 thresholds each, keeping the total "width" of the interface constant.
  • Binary Vision Transformers (BinaryViT): They temporarily expand the representation width to accommodate the extra threshold responses and then use a learned reduction layer to map it back to the original size.

Datasets

The researchers evaluated BiTE across several standard computer vision benchmarks:

  • CIFAR-10 & CIFAR-100: Used for primary evaluations on BinaryViT and ResNet-18.
  • ImageNet: Used for a preliminary large-scale study on BinaryViT.

Key Insights and Ideas

  • Rank Matters: By analyzing the "effective rank" of the network's internal representations, the authors found that BiTE produces significantly more diverse and less redundant information than methods that collapse their thresholds.
  • Smart Capacity Allocation: The paper shows that it is often better to have fewer feature directions with more "depth" (more thresholds) than many feature directions with only a single bit of information each.
  • Threshold Learning: The thresholds are not fixed; they are learned jointly with the rest of the network using a straight-through estimator (STE), allowing the network to adapt its "boundaries" to the data distribution.
Effective rank analysis showing that BiTE retains more information diversity than collapsed methods.

Key Results

  • Accuracy Boost: On BinaryViT, BiTE improved Top-1 accuracy from 74.60% to 75.83% on CIFAR-10 and from 42.80% to 44.07% on CIFAR-100, with almost no change in the number of trainable parameters.
  • Efficiency Gains: In VGG-16 experiments, one BiTE configuration achieved 0.64% higher accuracy while using 21.8% fewer parameters than a standard BNN scaled to a similar width.
  • Pareto Superiority: As shown in Figure 2, BiTE configurations consistently outperformed standard BNNs across various parameter budgets, effectively shifting the Pareto front of the accuracy-parameter trade-off.
Figure 2: Parameter–accuracy trade-off on VGG-16/CIFAR-10. (a) Highest test-set accuracy observed during training and (b) final-epoch test accuracy. Points report the mean over three runs, with error bars showing the sample standard deviation. Nondominance is determined from the mean parameter–accuracy coordinates among the evaluated configurations; connecting lines are visual guides only.

Limitations and Future Work

  • ImageNet Scaling: Preliminary results on ImageNet were mixed. While BiTE improved performance when applied to early layers, applying it throughout the entire network actually lowered accuracy. The authors suggest that placement and regularization strategies for large-scale models need further study.
  • Hardware Characterization: While the method is designed to be bitwise-friendly, the paper does not provide direct measurements of hardware latency, energy consumption, or memory traffic on specific edge devices.
  • Future Directions: The authors aim to evaluate BiTE on larger transformer backbones and conduct a deeper dive into its hardware-level efficiency.

Summary updated

Abstract

Binary neural networks (BNNs) reduce model storage and replace many arithmetic operations with efficient bitwise computations, but the single-sign activation limits their representational capacity. Each preactivation direction is compared with one threshold and reduced to one of two values, discarding variation within each sign region. We propose \textbf{BiTE} (Branch-Preserving Binary Threshold Expansion), which evaluates each feature direction against $K$ ordered thresholds and retains the resulting binary responses as separate inputs to the following learned operator. BiTE can distinguish up to $K+1$ ordered activation states while allowing the following operator to learn a separate transformation for each threshold response. It also enables a controllable trade-off between the number of independently constructed feature directions and the number of ordered responses extracted from each direction. On BinaryViT, BiTE improves mean final-epoch Top-1 accuracy from $74.60\pm0.30\%$ to $75.83\pm0.15\%$ on CIFAR-10 and from $42.80\pm0.46\%$ to $44.07\pm0.32\%$ on CIFAR-100, with nearly unchanged trainable parameter counts. A matched uniform-collapse control reaches $75.33\pm0.15\%$ on CIFAR-10, supporting the value of retaining the threshold responses separately relative to the evaluated scalar collapse. Binary VGG-16 experiments further show a favorable observed parameter--accuracy trade-off, including a configuration that achieves $0.64$ percentage points higher mean accuracy with $21.8\%$ fewer parameters than conventional width scaling. Overall, these results support branch-preserving threshold expansion as an effective way to improve binary representations and the observed parameter--accuracy trade-off.
BibTeX
@misc{gupta2026beyond,
  author = {Siddharth Gupta and Mikail Yayla and Akash Kumar},
  title = {{Beyond a Single Sign: BiTE for Branch-Preserving Binary Threshold Expansion}},
  year = {2026},
  eprint = {2611.17576},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17576},
}
Figure