Traditional single-image super-resolution (SISR) treats upscaling as a discrete grid-to-grid mapping problem. Architectures rely on fixed upsampling layers—such as sub-pixel convolutions or pixel-shuffle operations—that are hardcoded to specific integer magnification factors like $\times2$, $\times3$, or $\times4$. As noted by Xiaobiao Du et al., this design forces practitioners to train and deploy entirely separate models for every desired target scale.
This rigid constraint fails in real-world scenarios where magnification factors are continuous, non-integer, or highly unpredictable. For instance, clinical applications like brain MRI inspection often require arbitrary zooming to align varying slice thicknesses, as discussed by Chuan Tan et al. and Xin Wang et al.. Similarly, user-facing applications featuring pinch-to-zoom interactions or irregular geometric transformations cannot be constrained to regular rectangular grids, as demonstrated by Sanghyun Son et al..
From Grids to Continuous Functions
To break free from fixed-scale constraints, the field shifted from predicting discrete pixel grids to learning continuous coordinate representations. Instead of viewing an image as a matrix of pixels, we can treat it as a continuous function:
$$f(x, y) = I$$
where $(x, y)$ represents a continuous spatial coordinate and $I$ is the corresponding signal value, such as RGB color, depth, or tissue intensity.
This approach leverages Implicit Neural Representations (INR). An INR uses a neural network, typically a small Multi-Layer Perceptron (MLP), to parameterize this continuous function. Because a neural network can accept any real-valued coordinate as an input, we can query the function at an infinite resolution, enabling true arbitrary-scale upsampling.
The Mechanics of Local Implicit Querying
Implementing this continuous function using a low-resolution (LR) input image generally follows a three-step pipeline:
- Feature Encoding: The input LR image is passed through a standard convolutional or Transformer backbone to generate a dense feature map. This map retains the discrete grid structure but encodes rich local context.
- Coordinate Querying: For any desired high-resolution (HR) output pixel, its target coordinate $(x_q, y_q)$ is mapped into the coordinate space of the LR feature map.
- Relative Decoding: Instead of feeding the absolute coordinate directly to an MLP, the model identifies the nearest discrete feature vector or vectors in the LR grid. It computes the coordinate offset $(\Delta x, \Delta y)$ between the query point and the cell center. The MLP then takes the local LR feature vector and this relative offset as inputs to predict the final pixel value.
By operating on relative offsets rather than absolute coordinates, the MLP acts as a continuous local interpolator. It learns how features vary smoothly across space, allowing it to synthesize sharp, high-frequency details at any arbitrary scale factor.
Overcoming Artifacts via Meta Networks and Attention
While the basic INR pipeline works well, early implementations suffered from blocky artifacts near the boundaries of the discrete LR feature cells. Researchers have introduced several key mechanisms to improve reconstruction fidelity and smoothness.
Dynamic Weight Prediction
Rather than using a static MLP to decode features, some frameworks use a secondary network to dynamically predict the weights of the upsampling filters or the MLP itself based on the target scale factor and coordinate offsets. For example, Chuan Tan et al. utilize a Weight Prediction Network to adjust to non-integer scales dynamically. Similarly, Cheng Peng et al. employ a spatially aware meta-network to generate convolutional filters tailored to specific physical voxel spacings in medical scans.
Cross Scale Attention and Transformers
To eliminate grid artifacts entirely, recent architectures replace simple local interpolation with attention mechanisms. Instead of querying only the single nearest LR pixel, models attend to a neighborhood of features.
The authors of the Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolution introduced a cross-scale attention mechanism that aggregates information across multiple stages, preventing structural blurring. In a similar vein, Keyan Chen et al. use Transformers to model global context interaction within the implicit function space, ensuring that large-scale structures remain coherent during extreme upscaling.
Extending Beyond Two Dimensions
The beauty of the continuous coordinate paradigm is its mathematical generality. Because it treats signals as functions over a coordinate domain, it can easily be extended to higher dimensions or alternative data structures.
Video Super Resolution
By adding a time dimension $t$, the continuous function becomes $f(x, y, t)$. Models can perform simultaneous space-time super-resolution, upscaling both spatial resolution and frame rates to arbitrary values, as shown by Dongwei Ren et al.. To guide this complex spatiotemporal interpolation, the authors of Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution and the creators of EvEnhancer fuse standard RGB frames with high-speed data from event cameras within a unified implicit representation.
Depth Maps and 3D Point Clouds
For depth map super-resolution, Xiaohang Wang et al. treat depth as a continuous coordinate-learning problem to preserve sharp geometric boundaries. Moving to pure 3D data, Wanquan Feng introduces a paradigm shift in Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling, representing discrete 3D points as centers of continuous local surface patches parameterized by neural fields, enabling arbitrary-factored 3D upsampling.