CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture

Ruofan Liang, Hongyi Sun, Nandita Vijaykumar

Overview

CoordX is a novel Multi-Layer Perceptron (MLP) architecture designed to significantly accelerate Implicit Neural Representations (INRs). By processing coordinate dimensions (such as $x, y,$ and $z$) independently before fusing them, this "split" approach achieves up to a 2.9x speedup in training and inference across images, videos, and 3D shapes, all while maintaining high reconstruction accuracy.

The Computational Tax of Implicit Neural Representations

Implicit Neural Representations have revolutionized how we store and render digital media. Instead of saving an image as a discrete grid of pixels, an MLP is trained to act as a continuous function: input a spatial coordinate $(x, y)$, and it outputs a color. This paradigm allows for infinite resolution and high memory efficiency.

However, this comes with a massive "computation tax." To render a single $1024 \times 1024$ image, the MLP must be queried over a million times. For 3D scenes (like NeRFs), this cost explodes, often requiring hours of training and seconds per frame for inference. While previous solutions have utilized complex data structures like octrees to skip "empty space," they often add memory overhead or are highly task-specific. CoordX tackles this bottleneck directly at the architectural level, aiming for a general-purpose speedup.

The CoordX Architecture: Splitting the Bill

The core innovation of CoordX is its Split MLP Architecture.

In a standard baseline model (coordMLP), every single coordinate point is treated as an independent input vector. CoordX observes that points along the same axis inherently share structural information.

Comparison of the baseline coordMLP and the proposed CoordX architecture, highlighting the coordinate decomposition process.

The architecture optimizes processing through the following steps:

  1. Input Decomposition: Instead of feeding $(x, y)$ together, the model splits the input. One branch processes the $x$-coordinates, and another processes the $y$-coordinates.
  2. Split & Shared Layers: The initial layers are split into parallel branches. To keep the parameter count low, many of these layers are shared across branches, allowing the model to learn correlations between dimensions without doubling the weights.
  3. Feature Fusion (The Outer Product): To reconstruct the full signal, CoordX uses an outer product to fuse the features from the separate branches. This mathematically mirrors how a matrix can be decomposed into the product of two vectors (similar to Singular Value Decomposition), allowing the network to learn a compressed representation of the signal.
Visualizing the feature fusion process: the outer product of decomposed matrices reconstructs the final 2D image.
  1. Final Refinement: A few standard fully connected layers process the fused features to produce the final output (e.g., RGB color or 3D occupancy).

Because the initial layers process $H + W$ inputs instead of $H \times W$ inputs, the computational load drops dramatically.

Core Insights

  • Dimensional Locality: Standard MLPs ignore the fact that neighboring pixels share coordinate values. By splitting the architecture, CoordX explicitly leverages this locality, treating an image more like a decomposable matrix than a flat list of isolated points.
  • The Efficiency of Sums vs. Products: For an $H \times W$ image, CoordX reduces the primary input processing from $O(H \times W)$ to $O(H + W)$. As resolution increases, the speedup becomes exponentially more pronounced.
  • Orthogonality: CoordX is an architectural change, not a data-structure change. This means it can be seamlessly combined with other optimization techniques (like sparse octrees or multi-scale decompositions) to stack even more performance gains.

Performance and Results

The authors evaluated CoordX across several standard benchmarks, including 2D RGB images (DIV2K), 10-second video clips, 3D shape occupancy networks (Stanford 3D Scanning Repository), and Novel View Synthesis (Blender scenes).

  • Speedups: The architecture demonstrated a speedup of up to 2.92x for image tasks and 2.81x for 3D shape rendering.
  • NeRF Acceleration: In a simplified NeRF model, CoordX achieved a 1.81x training speedup and a staggering 4.41x rendering speedup with less than a 0.5 dB drop in quality.
  • Accuracy Retention: While there is a slight "accuracy tax" (roughly a 3dB PSNR drop in some high-frequency image tasks), this can be mitigated by increasing the "reduction dimension" ($R$), allowing users to trade a fraction of speed for higher fidelity.
  • Memory: Unlike many acceleration methods, CoordX introduces no additional memory overhead.

Limitations and Future Directions

While highly effective for multi-dimensional data, CoordX provides no benefit for unidimensional signals (like audio) because there are no dimensions to split. Additionally, the decomposition approach can sometimes struggle to capture extremely fine, high-frequency details in images compared to a "brute-force" MLP.

Furthermore, standard NeRF uses irregular ray sampling. To make CoordX work in these environments, the authors utilized a grid-interpolation workaround. Future work could involve making the split architecture more natively flexible for non-grid-aligned data and exploring its application in even higher-dimensional data modeling beyond 3D and video.

Summary updated

Abstract

Implicit neural representations with multi-layer perceptrons (MLPs) have recently gained prominence for a wide variety of tasks such as novel view synthesis and 3D object representation and rendering. However, a significant challenge with these representations is that both training and inference with an MLP over a large number of input coordinates to learn and represent an image, video, or 3D object, require large amounts of computation and incur long processing times. In this work, we aim to accelerate inference and training of coordinate-based MLPs for implicit neural representations by proposing a new split MLP architecture, CoordX. With CoordX, the initial layers are split to learn each dimension of the input coordinates separately. The intermediate features are then fused by the last layers to generate the learned signal at the corresponding coordinate point. This significantly reduces the amount of computation required and leads to large speedups in training and inference, while achieving similar accuracy as the baseline MLP. This approach thus aims at first learning functions that are a decomposition of the original signal and then fusing them to generate the learned signal. Our proposed architecture can be generally used for many implicit neural representation tasks with no additional memory overheads. We demonstrate a speedup of up to 2.92x compared to the baseline model for image, video, and 3D shape representation and rendering tasks.
BibTeX
@inproceedings{liang2022coordx,
  author = {Ruofan Liang and Hongyi Sun and Nandita Vijaykumar},
  title = {{CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=oAy7yPmdNz},
}
Figure