Implicit Neural Representations (INRs) have fundamentally changed how we store and manipulate continuous signals. Instead of saving an image as a discrete grid of pixels or a 3D shape as a dense matrix of voxels, an INR parameterizes the signal as a continuous function, $f_\theta(\mathbf{x}) = \mathbf{y}$, mapped by a neural network (typically a Multi-Layer Perceptron, or MLP).
This approach offers remarkable benefits: infinite resolution independence, smooth interpolation, and highly compact storage footprint. However, these advantages come with a severe, structural trade-off: coordinate querying overhead. To understand why this bottleneck exists, we must look at how data retrieval changes when moving from explicit memory arrays to implicit neural weights.
From Memory Lookups to Neural Evaluations
In a traditional discrete representation, retrieving a pixel value or a voxel occupancy is an $O(1)$ operation. The data exists explicitly in memory; the hardware simply calculates an array offset and reads the value.
Traditional Grid: [Coordinate x,y] ---> (O(1) Memory Lookup) ---> [Color/Value]
Implicit Field: [Coordinate x,y] ---> [MLP Forward Pass] ---> [Color/Value]
When Zhiqin Chen et al. introduced continuous implicit fields for generative shape modeling in Learning Implicit Fields for Generative Shape Modeling, they demonstrated that a network could predict whether any arbitrary 3D coordinate was inside or outside a shape. Yet, because the data is stored implicitly within the network weights $\theta$, you cannot simply "read" a pixel. To find out what exists at coordinate $\mathbf{x}$, you must execute a full forward pass through the network.
This architectural shift transforms a cheap memory-bandwidth problem into an expensive compute-bound problem. As a result, generating high-resolution outputs requires evaluating a dense grid of points from scratch, which becomes computationally prohibitive.
The Per-Pixel Scaling Bottleneck
The core reason INRs struggle with real-time performance is that their inference cost scales linearly with the number of target samples. If you want to render a high-definition image of size $H \times W$, you must perform $H \times W$ independent forward passes through the MLP.
As Minsu Kim et al. note in Implicit Neural Image Stitching With Enhanced and Blended Feature Reconstruction, this linear dependency on the total number of pixels creates a massive computational bottleneck compared to traditional grid-based transformations or standard convolutional layers. In a standard convolutional neural network (CNN), a single forward pass processes an entire feature map in parallel, leveraging spatial weight sharing. In contrast, an INR evaluates every coordinate point-by-point.
This issue is magnified in view synthesis and 3D rendering. Suttisak Wizadwongsa et al. point out in NeX: Real-Time View Synthesis With Neural Basis Expansion that models like NeRF require querying a deep neural network hundreds of times per ray for every single pixel. This high per-pixel evaluation cost prevents pure coordinate-based models from achieving the interactive frame rates required for real-time applications.
The Dimensionality Curse
The coordinate querying penalty compounds aggressively as we add dimensions. When moving from 2D images to 3D volumes, 4D videos, or hyperspectral data, the number of coordinates required to reconstruct the signal grows exponentially.
Consider the following domains where this bottleneck becomes critical:
- Medical Imaging (3D/4D): In Self-supervised arbitrary scale super-resolution framework for anisotropic MRI, Haonan Zhang et al. explain that executing a network query for every individual coordinate in a high-resolution 3D grid scales poorly with increasing volume size, limiting real-time clinical utility. When a temporal dimension is added—such as in the 4D fetal brain atlases constructed by Lixuan Chen et al. in Continuous longitudinal fetus brain atlas construction via implicit neural representation—the dense spatiotemporal sampling demands massive computational resources, restricting scalability.
- Hyperspectral Imaging: Reconstructing continuous spectral bands means adding a spectral dimension to the spatial coordinates. Ruikang Xu et al. demonstrate in Continuous Spectral Reconstruction from RGB Images via Implicit Neural Representation that evaluating an MLP for every combined spatial and spectral coordinate introduces severe inference latency compared to single-pass feed-forward convolutional architectures.
Shifting Paradigms for Faster Inference
To overcome the latency of point-wise MLP queries, researchers are actively developing alternative architectures that bridge the gap between continuous representations and fast, grid-like lookups.
1. Direct Grid Optimization
One way to eliminate the deep MLP bottleneck is to return to explicit data structures but keep them continuous via interpolation. Cheng Sun et al. proposed Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction, which replaces deep networks with directly optimized dense voxel grids. By utilizing fast trilinear interpolation over an explicit grid rather than evaluating a deep network hundreds of times per ray, they cut training and reconstruction times from days down to minutes.
2. Explicit Continuous Primitives (Gaussian Splatting)
Instead of querying an implicit function to find what exists at a coordinate, why not project explicit, continuous shapes directly onto the output space? This is the intuition behind adapting Gaussian Splatting to replace INRs.
Xinjie Zhang et al. introduced GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting, demonstrating that representing images via 2D Gaussians bypasses the heavy GPU memory and computational costs of MLPs. Building on this concept, Long Peng et al. developed Pixel to Gaussian: Ultra-Fast Continuous Super-Resolution with 2D Gaussian Modeling. By reconstructing a continuous Gaussian field in a single pass rather than relying on a per-pixel decoding mechanism, their framework achieves ultra-fast rendering speeds, unlocking real-time performance even as the target resolution scales.
3. Patch-Level Processing
Another strategy involves changing the granularity of the query. Instead of querying a single coordinate point, models can query localized regions. Yejia Zhang introduced SwIPE: Efficient and Robust Medical Image Segmentation with Implicit Patch Embeddings, which shifts the target from point-wise evaluation to patch-level shape prediction. This preserves the resolution flexibility of implicit representations while significantly reducing the total number of discrete network queries required to reconstruct high-resolution volumes.
Key Foundational Work
Implicit Neural Representations with Periodic Activation Functions (2020) — contributed 1 concept elements to this direction.
Learning Implicit Fields for Generative Shape Modeling. (2019) — contributed 1 concept elements to this direction.