Scaling Receptive Fields in Look-Up Table Inference

19 concepts16 papers1 proposal

Deep neural networks have revolutionized image restoration, but their computational cost remains a barrier for real-time deployment on mobile and edge hardware. A compelling alternative is the Look-Up Table (LUT), which replaces expensive convolutions with simple memory retrievals. By precomputing the output of a neural network for every possible input combination and storing it in a table, inference becomes a matter of indexing and interpolation.

However, this approach faces a fundamental scaling bottleneck: the "curse of dimensionality." As the number of input pixels (the receptive field) increases, the storage required for the table grows exponentially.

The Exponential Wall

The intuition behind the LUT bottleneck is straightforward. If a model considers a $2 \times 2$ neighborhood of 8-bit pixels, the table must account for $256^4$ (over 4 billion) possible input combinations. Adding just one more pixel to the receptive field multiplies the storage requirement by another factor of 256.

As noted by Jo et al. and Kim in Practical Single-Image Super-Resolution Using Look-Up Table, this exponential growth forces a severe trade-off. To keep memory footprints within the megabyte range suitable for mobile caches, traditional LUT methods are often restricted to tiny $2 \times 2$ or $3 \times 3$ receptive fields. This limitation prevents the model from capturing the long-range spatial dependencies and complex textures necessary for high-quality restoration, a problem highlighted across various domains from video coding (Wu et al.) to volumetric streaming (Wang et al.).

Decomposing the Receptive Field

To break this exponential curse, researchers have moved away from single, massive tables toward structured assemblies of smaller ones. The core idea is to mimic the architecture of a neural network using LUTs as the fundamental building blocks.

In MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution, Li et al. and the authors propose using multiple LUTs in parallel and cascade. Instead of one table indexing a large patch, several small tables index different subsets of pixels. Their outputs are then combined, effectively expanding the receptive field without the corresponding memory explosion. Similarly, Ma et al. and the authors of SPLUT utilize a series-parallel structure to aggregate spatial information, allowing the model to "see" more of the image while maintaining a manageable memory footprint.

Structural and Sampling Innovations

Beyond simple cascading, recent work has focused on how the input space is sampled and processed. Traditional LUTs use a rigid grid of pixels, but not all pixels in a neighborhood are equally informative.

Xu et al. and the authors of AutoLUT introduced Automatic Sampling, which learns to select the most relevant pixels for the LUT index rather than relying on a fixed local window. This allows the model to capture broader context with fewer input dimensions.

Another approach involves rethinking the mathematical operations being tabulated. In RCLUT, Liu et al. and the authors decouple spatial and channel-wise calculations into a "Reconstructed Convolution" module. By breaking down the convolution into smaller, independent components that can each be represented by a small LUT, they achieved a $27 \times 27$ receptive field—orders of magnitude larger than previous methods—while using only 1.5 MB of memory.

Exploiting Natural Image Redundancy

Finally, the efficiency of these models is further enhanced by recognizing that the input space of natural images is not uniformly distributed. In Look-Up Table Compression for Efficient Image Restoration, the authors introduce Diagonal-First Compression (DFC). This technique exploits "diagonal dominance"—the fact that adjacent pixels in natural images tend to have similar values. By prioritizing the storage of values near the diagonal of the input space, DFC can reduce LUT storage requirements by up to 90% without sacrificing restoration quality.

Through these combined strategies—cascading, learnable sampling, and structural decomposition—researchers have transformed the LUT from a limited local operator into a scalable framework capable of high-fidelity image and video restoration (Qu et al.) at speeds that traditional neural networks cannot match.

Key Foundational Work

Practical Single-Image Super-Resolution Using Look-Up Table (2021) — contributed 1 concept elements to this direction.
Semi-Global Weighted Least Squares in Image Filtering (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_exponential_memory_scaling_02b5393f,
  title = {exponential memory scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exponential-memory-scaling/rdr_02b5393f783d49f5b60f084c9a83cfce},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_exponential_memory_scaling_02b5393f_v1,
  title = {exponential memory scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exponential-memory-scaling/rdr_02b5393f783d49f5b60f084c9a83cfce/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake