CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture

Ruofan Liang, Hongyi Sun, Nandita Vijaykumar

Abstract

Implicit neural representations with multi-layer perceptrons (MLPs) have recently gained prominence for a wide variety of tasks such as novel view synthesis and 3D object representation and rendering. However, a significant challenge with these representations is that both training and inference with an MLP over a large number of input coordinates to learn and represent an image, video, or 3D object, require large amounts of computation and incur long processing times. In this work, we aim to accelerate inference and training of coordinate-based MLPs for implicit neural representations by proposing a new split MLP architecture, CoordX. With CoordX, the initial layers are split to learn each dimension of the input coordinates separately. The intermediate features are then fused by the last layers to generate the learned signal at the corresponding coordinate point. This significantly reduces the amount of computation required and leads to large speedups in training and inference, while achieving similar accuracy as the baseline MLP. This approach thus aims at first learning functions that are a decomposition of the original signal and then fusing them to generate the learned signal. Our proposed architecture can be generally used for many implicit neural representation tasks with no additional memory overheads. We demonstrate a speedup of up to 2.92x compared to the baseline model for image, video, and 3D shape representation and rendering tasks.

Introduction

Implicit Neural Representations (INRs) use coordinate-based MLPs to map positions (e.g., $x, y, z$) to signal values (e.g., color, occupancy). While powerful and memory-efficient compared to discrete grids, they are computationally expensive. To render a single $1024 \times 1024$ image, a standard MLP must be queried over a million times. This "coordinate bottleneck" makes high-resolution rendering and training painfully slow. CoordX addresses this by introducing a structural architectural change that exploits the grid-like nature of many coordinate-based tasks.

Methods: The Split MLP Architecture

The core innovation of CoordX is the decomposition of the input coordinate space. Instead of feeding a full coordinate vector (e.g., $[x, y]$) into one large MLP, CoordX splits the input into its constituent dimensions.

Overview of the CoordX architecture compared to standard coordMLP. The process involves decomposing coordinate inputs into separate vectors which are processed by split branches and then fused.

As shown in the architecture overview, the process consists of:

  1. Input Decomposition: Coordinates are separated into independent vectors (e.g., $x$ and $y$).
  2. Split Initial Layers: Parallel branches process these dimensions separately. These branches share weights to maintain a parameter count comparable to standard MLPs while capturing cross-dimensional correlations.
  3. Feature Fusion via Outer Product: The intermediate features from the split branches are combined using an outer product (multiplicative modulation). This allows the network to reconstruct the full multi-dimensional signal from its decomposed components.
  4. Final Refinement: A few standard fully connected layers generate the final output (e.g., RGB values).

By splitting the input, the number of operations in the initial layers drops from $O(H \times W)$ to $O(H + W)$. For high-resolution data, this represents a massive reduction in FLOPs.

Experimental Results

The authors evaluated CoordX across standard benchmarks including DIV2K images, high-resolution video clips, and the Stanford 3D Scanning Repository.

Inference time comparison between standard MLP and CoordX (Ours) across various image resolutions. CoordX demonstrates increasing speedup as resolution grows.

Key performance metrics include:

  • Inference Speedup: CoordX achieved a 2.35x to 2.92x speedup for image rendering and a 2.78x speedup for 1k resolution video clips. For a $256^3$ resolution 3D grid, it provided a 2.81x speedup.
  • Training Efficiency: For video tasks, CoordX was 2.02x faster to train than the baseline while maintaining similar quality.
  • Accuracy: In 3D shape representation, CoordX achieved nearly identical IoU (Intersection over Union) scores to the baseline. While a slight PSNR drop (~3dB) was noted in image fitting, the authors demonstrated this could be mitigated by increasing the "reduction dimension" ($R$) of the fusion layer.

Conclusion

CoordX provides a generalizable architectural template for accelerating INRs by leveraging coordinate decomposition and multiplicative fusion. While it is most effective for multi-dimensional signals and grid-aligned data, its compatibility with existing frameworks like NeRF and techniques like Positional Encoding makes it a versatile solution for reducing the computational burden of implicit representations. Future work will explore its application in neural implicit compression and modeling beyond visual signals.

Summary updated

Abstract

Implicit neural representations with multi-layer perceptrons (MLPs) have recently gained prominence for a wide variety of tasks such as novel view synthesis and 3D object representation and rendering. However, a significant challenge with these representations is that both training and inference with an MLP over a large number of input coordinates to learn and represent an image, video, or 3D object, require large amounts of computation and incur long processing times. In this work, we aim to accelerate inference and training of coordinate-based MLPs for implicit neural representations by proposing a new split MLP architecture, CoordX. With CoordX, the initial layers are split to learn each dimension of the input coordinates separately. The intermediate features are then fused by the last layers to generate the learned signal at the corresponding coordinate point. This significantly reduces the amount of computation required and leads to large speedups in training and inference, while achieving similar accuracy as the baseline MLP. This approach thus aims at first learning functions that are a decomposition of the original signal and then fusing them to generate the learned signal. Our proposed architecture can be generally used for many implicit neural representation tasks with no additional memory overheads. We demonstrate a speedup of up to 2.92x compared to the baseline model for image, video, and 3D shape representation and rendering tasks.
BibTeX
@inproceedings{liang2022coordx,
  author = {Ruofan Liang and Hongyi Sun and Nandita Vijaykumar},
  title = {{CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=oAy7yPmdNz},
}
Figure