Replacing Multiplications with L1 Norm Distance in Neural Networks

29 concepts20 papers1 proposal

The remarkable success of deep learning relies heavily on a single mathematical operation: the dot product. Whether in a convolutional layer or a fully connected layer, networks spend the vast majority of their computational budget performing cross-correlation, multiplying input features by learned weights.

From a hardware perspective, this is a massive bottleneck. Multipliers are computationally expensive, demanding significantly more silicon area and electrical power than simple adders. As deep learning moves onto resource-constrained edge devices, satellites, and wearable hardware, researchers have asked a fundamental question: Do we really need multiplications in deep learning?

A powerful alternative has emerged: replacing the standard inner product with the $\ell_1$-norm distance (or Manhattan distance). By reframing feature extraction as a distance-measuring problem rather than a projection problem, neural networks can eliminate power-hungry multiplications entirely, replacing them with hardware-efficient additions and subtractions.

The Intuition: Similarity via Distance

To understand why this substitution works, we must look at what a standard convolutional layer actually computes. When a filter slides across an image, the dot product measures the alignment between the filter weights $W$ and the input patch $X$:

$$Y = X \cdot W = \sum_{i} x_i \cdot w_i$$

If the input patch matches the pattern of the filter, the dot product yields a large positive value. The dot product acts as a metric of similarity.

However, we can measure similarity in other ways. Instead of asking "how well do these vectors align when multiplied?", we can ask "how close are these two vectors in space?". The $\ell_1$-norm distance measures exactly this:

$$D(X, W) = \|X - W\|_1 = \sum_{i} |x_i - w_i|$$

If the input patch $X$ is identical to the filter $W$, the distance is zero. As the patch becomes less similar to the filter, the distance increases. To turn this into a response that behaves like a traditional activation—where a perfect match produces the highest output—we can simply take the negative $\ell_1$-norm distance:

$$Y = -\|X - W\|_1$$

This mathematical shift completely changes the underlying hardware requirements. Computing $|x_i - w_i|$ requires only a subtraction and a conditional sign flip (absolute value). Summing these differences requires only additions. Multipliers are entirely eliminated from the forward pass.

The Hardware Advantage

The motivation for shifting from dot products to $\ell_1$-norm distances becomes clear when looking at hardware implementation. In digital circuits, a 32-bit floating-point multiplier requires vastly more logic gates and energy than a 32-bit adder.

This hardware asymmetry has been recognized since the early days of neural network hardware design. For instance, Melton et al. demonstrated in their VLSI Implementation of TInMANN that using Manhattan distance instead of Euclidean distance drastically simplified arithmetic logic units, allowing hundreds of neurons to fit onto a single digital chip. Similarly, in analog computing, Carvajal et al. showed in Subspace-Based Face Recognition in Analog VLSI that Manhattan distance can be implemented efficiently using current-mode logic, drastically reducing circuit complexity while maintaining classification accuracy.

For computer vision pipelines on FPGAs, Culurciello et al. demonstrated in Clustering Learning for Robotic Vision that substituting standard convolutions with a Sum of Absolute Differences (SAD) metric reduced silicon area and power consumption by up to 15 times, making real-time robotic vision viable on power-constrained hardware.

Overcoming the Optimization Bottleneck

If addition-based networks are so efficient, why did standard multiplication-heavy networks dominate the deep learning revolution? The answer lies in optimization.

When Chen et al. formalized modern addition-based deep networks in AdderNet: Do We Really Need Multiplications in Deep Learning?, they uncovered a major hurdle: $\ell_1$-norm distances do not behave like dot products during gradient descent.

In a standard convolutional layer, the gradient with respect to the input depends directly on the weights, and vice versa. This creates a natural scaling mechanism. In contrast, the derivative of the absolute value function $|x_i - w_i|$ is simply the sign function ($\pm 1$). The magnitude of the gradient is decoupled from the magnitude of the error, leading to severe training instability and poor convergence when training deep architectures from scratch.

Furthermore, the statistical distribution of the features changes. Standard convolutions tend to generate outputs that follow a Gaussian distribution. Because $\ell_1$-norm operations compute absolute differences, they naturally generate features that follow a Laplacian distribution. This distribution mismatch prevents addition-based networks from easily utilizing standard architectural components like Batch Normalization.

To resolve these optimization bottlenecks, researchers developed several key techniques:

  1. Distribution Matching via Distillation: To force the Laplacian features of an addition network to behave like the well-behaved Gaussian features of a standard CNN, Chen et al. introduced a specialized distillation framework in Kernel Based Progressive Distillation for Adder Neural Networks. By mapping different feature distributions into a shared high-dimensional space, a multiplication-heavy teacher network can guide the addition-based student network to learn robust representations.
  2. Adaptive Normalization: To stabilize the gradient flow, Dong et al. proposed Adaptive Weight Normalization (AWN) in Towards Stable and Robust AdderNets. They discovered that when properly normalized, addition networks are not only stable but also inherently more robust to adversarial attacks than traditional CNNs.
  3. Architectural Adjustments: For complex tasks like object detection, Xu showed in An Empirical Study of Adder Neural Networks for Object Detection that standard feature aggregation paths fail due to the variance differences between layers. Introducing specialized path aggregations and revised normalization strategies allowed addition-based networks to match standard models while cutting energy consumption by 1.4x.

Scaling Across Domains

Once the optimization challenges were resolved, the addition-based paradigm proved highly versatile, expanding far beyond basic image classification into a wide array of energy-critical applications.

Image and Video Processing

In high-resolution image processing, where the sheer volume of pixels amplifies computational costs, addition-based networks offer massive savings. Song et al. developed AdderSR: Towards Energy Efficient Image Super-Resolution, which replaced multiplications with $\ell_1$-norm distances to achieve a 2.5x reduction in energy consumption during image upscaling. For dense prediction tasks, Yin et al. designed an efficient salient object detection network in Exploring Salient Object Detection with Adder Neural Networks by introducing a differential merging strategy to preserve edge details without relying on multipliers.

3D and Spatial Computing

The paradigm has also been extended to three-dimensional data. In Rethinking 3D Convolution in $\ell_p$-norm Space, Zhang et al. replaced traditional inner-product 3D convolutions with $\ell_1$-norm distance calculations for point cloud processing, reducing theoretical hardware energy consumption by over 60% and lowering instruction latency. Meanwhile, for mobile hardware like AR glasses, Liu et al. demonstrated in Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses that replacing complex per-pixel similarity metrics with hardware-optimized group-pointwise convolutions and layer normalization significantly cuts computational latency.

Audio and Embedded Sensing

For always-on edge devices, eliminating multiplications is a game-changer. In Energy-Friendly Keyword Spotting System Using Add-Based Convolution, Chen et al. eliminated up to 99% of multiplication operations in a keyword spotting system with less than a 1% drop in accuracy. Similarly, Badawi et al. utilized an additive network architecture (AddNet) in Detecting Gas Vapor Leaks Using Uncalibrated Sensors to deploy deep learning models directly onto resource-constrained gas sensors.

Advanced Hardware Paradigms

To push efficiency even further, researchers are combining addition-based networks with other compression and hardware paradigms:

By shifting the fundamental mathematical primitive of deep learning from multiplication to addition, this research direction proves that the high energy footprint of modern AI is not an inevitability, but an artifact of how we choose to measure similarity. Reframing our algorithms around hardware-efficient distance metrics paves the way for a new generation of ubiquitous, low-power intelligent devices.

Key Foundational Work

VLSI Implementation of TInMANN (1990) — contributed 2 concept elements to this direction.
Subspace-Based Face Recognition in Analog VLSI (2007) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_additive_feature_extraction_dd5b48fc,
  title = {additive feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/additive-feature-extraction/rdr_dd5b48fc98a54114ba90538b90990b8d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_additive_feature_extraction_dd5b48fc_v1,
  title = {additive feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/additive-feature-extraction/rdr_dd5b48fc98a54114ba90538b90990b8d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake