Linear Feature Synthesis for Efficient Neural Networks

This direction is split into other directions: feature map redundancy exploitation. Its last version stays readable and citable here.

Version 1 of 179 papers119 conceptsfrozen 2026-10-04

Deep convolutional neural networks (CNNs) are notoriously resource-hungry, often requiring millions of parameters and billions of floating-point operations (FLOPs) to achieve state-of-the-art accuracy. However, if you look closely at the intermediate feature maps of a trained network, you will notice a striking pattern: many maps are nearly identical. These "redundant" features are not necessarily a waste of space; they provide the representational diversity the network needs to generalize.

The core insight of this research direction is that we don't need to compute every one of these maps from scratch using expensive convolutions. Instead, we can generate a small set of "intrinsic" features and then synthesize the rest using "cheap" linear operations.

The Ghost Module Mechanism

The fundamental building block of this approach is the Ghost module, introduced by Kai Han, Yunhe Wang, and their colleagues in GhostNet: More Features from Cheap Operations.

In a standard convolutional layer, every output feature map is the result of a full-rank convolution across all input channels. This is computationally expensive. A Ghost module replaces this with a two-step process:

  1. Intrinsic Feature Generation: A standard convolution produces a small number of primary feature maps. For example, if you need 64 output maps, you might only compute 32 using standard kernels.
  2. Cheap Linear Augmentation: The remaining 32 maps are generated by applying simple, computationally inexpensive linear transformations—such as depthwise convolutions or spatial shifts—to the intrinsic maps.
  3. Concatenation: The intrinsic maps and the "ghost" maps are concatenated to form the final output.

This strategy exploits the inherent redundancy in neural network representations. As Zewen Li and Wenjie Yang note in their survey of CNNs, treating feature redundancy as a computational opportunity rather than a waste allows networks to maintain high-dimensional feature spaces while drastically reducing memory access and parameter counts.

Why Linear Synthesis Works

The intuition behind linear synthesis is that the relationship between redundant feature maps can often be approximated by a simple linear function. If feature map $B$ is just a slightly blurred or shifted version of feature map $A$, it is far more efficient to learn a small $3 \times 3$ depthwise filter to transform $A$ into $B$ than it is to learn an entirely new set of weights to generate $B$ from the previous layer's input.

This approach has proven remarkably robust across different hardware. In GhostNets on Heterogeneous Devices via Cheap Operations, Kai Han demonstrated that these modules can be tailored for both CPUs (C-Ghost) and GPUs (G-Ghost), optimizing for the specific bottlenecks of each architecture. On mobile devices, where memory bandwidth and power are limited, reducing the number of "heavy" convolutions leads to significant latency improvements without the accuracy drop typically seen in aggressive pruning.

Beyond Standard Classification

While originally designed for image classification, the principle of cheap feature synthesis has expanded into specialized domains:

  • Image Super-Resolution: Chuanjian Liu and Kai Han developed GhostSR, which uses learnable shift operations to generate ghost features. This reduced GPU latency by up to 42% in super-resolution tasks, where high-resolution feature maps are particularly expensive to compute.
  • Natural Language Processing: The concept isn't limited to spatial data. Zhiqi Huang and Lu Hou introduced GhostBERT, applying ghost modules to the linear projections within Transformer blocks. They found that redundant features in attention mechanisms could also be synthesized, allowing BERT models to maintain performance while cutting computational costs in half.
  • Remote Sensing and IoT: In fields like satellite imagery and IoT-based mask detection, researchers like Tao Lei and Cheng Huang have used ghost-based architectures (Difference Enhancement and Spatial–Spectral Nonlocal Network) to enable real-time processing on edge devices where standard heavy backbones would fail.

Redundancy as a Design Principle

The success of this direction suggests a shift in how we think about model compression. Traditional methods like pruning or quantization often view redundancy as something to be removed. In contrast, linear feature synthesis views redundancy as a structural necessity that should be simulated rather than eliminated.

By using a small set of "seed" features to derive a larger population of maps, as explored by Chuntao Ding in GroupNL, researchers are finding that we can achieve the representational power of deep, wide networks with the footprint of much smaller ones. The "ghosts" in the network are not just remnants; they are the key to making high-performance AI viable on the edge.

Papers in this version

Showing 60 of 79 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_feature_synthesis_for_efficient_n_17dab69a,
  title = {Linear Feature Synthesis for Efficient Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-feature-synthesis-for-efficient-neural-networks/rdr_17dab69ac9ba4b979f8a01161734184d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_feature_synthesis_for_efficient_n_17dab69a_v1,
  title = {Linear Feature Synthesis for Efficient Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-feature-synthesis-for-efficient-neural-networks/rdr_17dab69ac9ba4b979f8a01161734184d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}