Reducing Redundant Feature Computation in Patch Based Inference

113 concepts105 papers1 proposal

When deep learning models process high-resolution images, they often face a memory wall. A common solution is to break the image into smaller "patches" or "tiles" and process them independently. However, this convenience comes with a hidden cost: the redundancy tax. To ensure smooth transitions between patches and maintain a valid receptive field, these patches must overlap. Consequently, the model ends up performing the same mathematical operations on the same pixels multiple times.

The Problem of Overlapping Context

The root of computational redundancy lies in the spatial requirements of convolutional and attention-based layers. As Lin et al. explain, even when optimizing for tiny microcontrollers, overlapping regions must be processed repeatedly to maintain the necessary receptive fields. If a model processes patches in total isolation, "tiling effects" or visible seams appear at the boundaries.

To mitigate these artifacts, researchers traditionally use dense overlapping and weighted blending. Wang et al. note that while this ensures visual continuity, it introduces significant computational overhead. In tasks like image restoration, Özdenizci and Legenstein describe a fundamental tradeoff: increasing patch overlap improves reconstruction smoothness but linearly increases the number of inference passes required for a single image.

The Mechanics of Redundancy

Why is this so inefficient? Consider a sliding window moving across an image. If the window moves one pixel at a time (a stride of 1), a single pixel in the center of the image will be "seen" by the model as many times as there are pixels in the window. Giusti et al. observed that this traditional patch-by-patch approach causes the same convolution and pooling operations to be repeated millions of times.

This inefficiency scales poorly. In the context of high-definition synthesis, Zheng and Deng argue that explicit overlap approaches result in slow inference speeds that hinder practical 4K image generation. Similarly, Fan highlights that in panoramic object detection, processing multiple overlapping sub-projections to handle geometric distortion creates a bottleneck for real-time performance.

Strategies for Eliminating Redundancy

Researchers have developed several distinct strategies to reclaim this wasted computation:

1. Fully Convolutional Reformulation Instead of treating patches as independent inputs, models can be designed to process the entire image (or much larger tiles) in a single pass. Oquab et al. suggest that transitioning to fully convolutional methods reduces the overhead of multi-scale sliding windows by sharing features across the entire spatial extent. Giusti et al. demonstrated that moving away from redundant patch-based segmentation could achieve speedups of up to 1500x.

2. Hidden State Caching In autoregressive models, where each new sample depends on previous ones, the redundancy is temporal. Ramachandran introduced a caching strategy that stores previously computed hidden states. By reusing these states instead of recalculating the entire receptive field for every new sample, the authors transformed the generation complexity from exponential to linear.

3. Delayed Aggregation and Reordering In 3D point cloud processing, points often inhabit multiple overlapping neighborhoods. Feng et al. proposed "delayed-aggregation," which extracts features from individual points before grouping them. This ensures that each point's features are calculated only once, regardless of how many neighborhoods it belongs to, eliminating massive multiply-accumulate redundancies.

4. Adaptive and Sparse Inference Not all patches require the same amount of work. Wang et al. developed Adaptive Patch Exiting (APE), allowing "easy" patches to exit the network early. In the generative space, Yu et al. introduced FISEdit, which uses a cache-enabled sparse diffusion process to only re-calculate the specific regions of an image affected by a text change, rather than re-rendering the entire frame.

The Path Toward Efficiency

The shift from independent patch processing to integrated feature reuse represents a maturing of computer vision architectures. Early milestones like R-CNN were criticized by Girshick et al. for the high computational redundancy of performing independent forward passes on thousands of overlapping region proposals. Modern approaches, such as the "Codemaps" proposed by Li et al., prioritize moving classification steps before global pooling to allow for the nearly instantaneous evaluation of overlapping regions.

By treating the image as a continuous field of information rather than a collection of isolated blocks, these techniques allow models to scale to higher resolutions and more constrained hardware without paying the "redundancy tax."

Key Foundational Work

A Fast Learning Algorithm for Image Segmentation with Max-Pooling Convolutional Networks (2013) — contributed 1 concept elements to this direction.
Fast Image Scanning with Deep Max-Pooling Convolutional Neural Networks (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_redundant_feature_computation_37c05433,
  title = {redundant feature computation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/redundant-feature-computation/rdr_37c054336f524ddba72a2c23d76361fe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_redundant_feature_computation_37c05433_v1,
  title = {redundant feature computation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/redundant-feature-computation/rdr_37c054336f524ddba72a2c23d76361fe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake