Every forward pass in a modern deep neural network is fundamentally an exercise in heavy arithmetic. Layers compute massive matrix-vector multiplications, convolutions, and non-linear activations over and over again. However, in many practical applications, the inputs to these layers—whether they are discrete vocabulary tokens in a language model, local pixel patches in an image, or static prompt contexts—belong to a finite or bounded space.
This observation underpins a powerful paradigm for efficiency: network-to-LUT conversion and embedding caching. Instead of repeatedly executing expensive floating-point operations (FLOPs) at runtime, we can precompute the outputs of neural network layers offline and store them in Look-Up Tables (LUTs) or static caches. At inference time, the model bypasses active tensor computation entirely, replacing it with constant-time memory fetches ($O(1)$ lookups) or simple vector additions.
The Core Intuition: Shifting Compute to Memory
To understand why this works, consider the very first layer of a standard neural network language model. The input is a discrete token from a fixed vocabulary, represented as a one-hot vector $x$. The layer computes a matrix multiplication $W \cdot x$ to produce a hidden representation. Because $x$ can only take a finite number of discrete states, the result of $W \cdot x$ for every possible token is entirely deterministic and known immediately after training.
By pre-calculating these outputs, the initial matrix multiplication transforms into a simple table lookup. As Jacob Devlin et al. demonstrated in Fast and Robust Neural Network Joint Models for Statistical Machine Translation, pre-computing the dot products between input embeddings and hidden layer weights transforms the initial forward pass from an expensive matrix operation into a series of rapid vector additions.
Standard Inference:
Input Token ──> [ Embedding Lookup ] ──> [ Matrix Multiplication (W × x) ] ──> Hidden State
Tabularized Inference:
Input Token ──> [ Precomputed Lookup Table (LUT) ] ──────────────────────────> Hidden State
This strategy shifts the computational burden from the processor to the memory subsystem. It allows deep, highly accurate models to achieve the real-time throughput of shallow networks by trading a larger memory footprint (RAM/VRAM) for a drastic reduction in runtime latency.
Tabularizing Text and Sequence Models
While caching the very first embedding layer is straightforward, researchers have extended this intuition much deeper into sequence architectures.
Skipping Shallow Layers
Instead of caching single tokens, we can cache sequences. Jue WANG et al. introduced this concept to large language models with SkipBERT: Efficient Inference with Shallow Layer Skipping. They replaced the computationally expensive lower layers of BERT with a Precomputed Lookup Table (PLOT) containing representations for common n-gram sequences. Because local context in shallow layers remains highly stable, retrieving these precomputed n-gram vectors allows the model to skip the bottom layers entirely without sacrificing downstream accuracy.
Decoupling Feed-Forward Networks
In standard Transformers, the Feed-Forward Networks (FFNs) are interleaved with self-attention layers, making their inputs highly dependent on the entire conversation history. To make FFNs pre-computable, Ajay Jaiswal et al. developed MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers. By completely decoupling the FFNs from the contextual residual stream, they allowed the FFNs to operate as deterministic, token-indexed lookup tables. The outputs for the entire vocabulary can be pre-calculated and stored on disk, loading them into active memory only when specific tokens appear in the input sequence.
Caching Static Contexts
In scenarios like in-context learning or multi-domain deployment, parts of the input sequence remain static across multiple queries. Qinyuan Ye et al. exploited this in FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning by encoding few-shot demonstration examples independently of the target query. This allows their hidden representations to be computed once and cached, meaning the model only needs to encode the new query at runtime. Similarly, Saket Dingliwal et al. utilized this mechanism in Domain Prompts: Towards memory and compute efficient domain adaptation of ASR systems, pre-computing and caching the activations of static domain-specific prefixes to eliminate latency overhead during speech rescoring.
Tabularizing Vision and Continuous Fields
The application of lookup tables is not restricted to discrete text tokens. In computer vision, input spaces like pixel intensities are naturally bounded (e.g., 0–255), making them prime candidates for tabularization.
For instance, in image super-resolution, Younghyun Jo et al. proved in Practical Single-Image Super-Resolution Using Look-Up Table that complex deep learning computations can be entirely replaced at inference time. By precalculating the outputs of a deep network for all possible local input patches and storing them in a look-up table, they enabled high-fidelity image upscaling via constant-time memory retrievals, running faster than standard mathematical interpolation on low-power mobile hardware.
To handle continuous coordinate spaces and arbitrary warping, Jiacheng Li developed Learning Steerable Function for Efficient Image Resampling. The proposed method pre-calculates the outputs of a lightweight neural network for various local patterns and stores them in look-up tables. During inference, the system retrieves optimized interpolation parameters via memory lookups instead of executing neural network layers, bringing the speed of traditional mathematical interpolation to sophisticated learned priors.
Overcoming the Curse of Dimensionality
If tabularizing neural networks is so effective, why not convert every layer into a lookup table? The primary obstacle is the exponential curse of dimensionality. If a lookup table takes an input of dimension $N$, where each dimension can take $K$ discrete values, the total size of the table scales as $O(K^N)$. For high-dimensional hidden states, storing a single naive lookup table would quickly require petabytes of memory.
To bypass this bottleneck, researchers use two primary strategies:
- Hierarchical and Parallel LUTs: Instead of mapping a large input space to an output using a single massive table, networks can be broken down into smaller, interacting tables. The authors of Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables structured multiple look-up tables in hierarchical and parallel configurations. This architectural shift allows table-based methods to achieve a significantly larger receptive field and match the quality of deep networks without experiencing an exponential explosion in table size.
- Indexing and Interpolation: Rather than storing every exact coordinate, models store a coarser grid and use fast interpolation techniques at runtime. Sidi Yang et al. leveraged this in Taming Lookup Tables for Efficient Image Retouching, demonstrating that substituting convolutional layers with purely lookup-table-based architectures combined with lightweight indexing and interpolation allows complex image enhancement to run in under a millisecond on standard CPUs.
By reframing neural network execution as a hybrid of memory retrieval and sparse interpolation, this research direction provides a practical path toward deploying state-of-the-art AI capabilities on highly resource-constrained edge devices, mobile processors, and real-time systems.
Key Foundational Work
Pre-Computable Multi-Layer Neural Network Language Models (2015) — contributed 3 concept elements to this direction.
High-order Graph-based Neural Dependency Parsing (2015) — contributed 2 concept elements to this direction.
A Fast and Accurate Dependency Parser using Neural Networks (2014) — contributed 2 concept elements to this direction.