Multiplication Free Neural Networks via L1 Distance

This direction is merged into another direction: additive feature extraction. Its last version stays readable and citable here.

Version 1 of 114 papers17 conceptsfrozen 2026-10-04

In standard deep learning, the "workhorse" operation is the convolution, which relies heavily on the dot product. To compute a single feature, a hardware processor must perform thousands of floating-point multiplications. However, in the world of hardware design, multiplications are expensive: they consume significantly more silicon area and power than simple additions.

A growing body of research, centered around the concept of Adder Neural Networks (AdderNets), proposes a radical shift: replacing these power-hungry multiplications with additions by rethinking how we measure the similarity between a filter and an input.

Similarity as Distance

The intuition behind a standard convolutional layer is that the dot product measures the correlation between a filter $w$ and an input patch $x$. If the patterns match, the output is high. Hanting Chen, Yunhe Wang, and their colleagues at Huawei's Noah's Ark Lab proposed in AdderNet: Do We Really Need Multiplications in Deep Learning? that we can capture this same "matching" logic using the $\ell_1$-norm (Manhattan distance).

Instead of computing $y = \sum (x \cdot w)$, an AdderNet computes: $$y = -\sum |x - w|$$

In this formulation, if the input $x$ is very close to the filter $w$, the distance is small, and the output (being negative) is large. This effectively transforms the feature extraction process into a distance-based similarity check. Because calculating the $\ell_1$-norm only requires subtraction and absolute value operations—both of which are implemented using addition-based logic—the need for floating-point multipliers is eliminated.

The Optimization Challenge

While the hardware benefits are clear, training these networks is not straightforward. In a standard CNN, the gradient of the output with respect to the weights is linear. In an AdderNet, the derivative of the absolute difference $|x - w|$ is the sign function ($\pm 1$). This leads to two major problems:

  1. Gradient Magnitude: The gradients do not scale with the magnitude of the error in the same way, making optimization unstable.
  2. Feature Distribution: Standard CNN features tend to follow a Gaussian distribution, while AdderNet features follow a Laplacian distribution.

To solve this, Hanting Chen and Chang Xu introduced specialized training techniques in Adder Neural Networks. They utilized an "adaptive learning rate" for different layers to balance the gradient magnitudes and employed Kernel Based Progressive Distillation (PKKD). PKKD uses a standard CNN as a "teacher" to help the "student" AdderNet learn to map its Laplacian features into a space that mimics the more discriminative Gaussian distributions of traditional networks.

Beyond Image Classification

The shift to addition-only operations has proven versatile across different domains:

  • 3D Vision: Li Zhang and Liu Liu extended this logic to point clouds in Rethinking 3D Convolution in $\ell_p$-norm Space, achieving a 61% reduction in theoretical energy consumption for 3D tasks.
  • Super-Resolution: In AdderSR, Dehua Song showed that by using "self-shortcuts" to compensate for the lack of high-frequency information in distance-based features, AdderNets could perform image upscaling with 2.5x less energy.
  • Object Detection: Xinghao Chen and Minjing Dong addressed the instability of training deeper detection models by proposing Reverse Path Aggregation, allowing AdderNets to match ResNet-level performance on COCO benchmarks.

Hardware Realization

The ultimate goal of this research is deployment on resource-constrained edge devices. Early work by Eugenio Culurciello and Clement Farabet in Clustering Learning for Robotic Vision demonstrated that distance-based metrics could significantly reduce the silicon area required for robotic vision.

More recently, Yunhe Wang and Mingqiang Huang validated these theoretical gains on physical hardware. In AdderNet and its Minimalist Hardware Design, they showed that an FPGA implementation of an AdderNet reduces power consumption by 78% and circuit area by 71% compared to a standard CNN. By removing the multiplier—the most "expensive" component of a digital neuron—AdderNets provide a path toward high-performance AI that can run on the smallest of batteries.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multiplication_free_neural_networks_via__dc808856,
  title = {Multiplication Free Neural Networks via L1 Distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiplication-free-neural-networks-via-l1-distance/rdr_dc80885699144da6bd684bc4dd60ec71},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multiplication_free_neural_networks_via__dc808856_v1,
  title = {Multiplication Free Neural Networks via L1 Distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiplication-free-neural-networks-via-l1-distance/rdr_dc80885699144da6bd684bc4dd60ec71/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}