When neural scene representations like Neural Radiance Fields (NeRF) emerged, they presented a stark trade-off between memory and speed. On one end, coordinate-based Multi-Layer Perceptrons (MLPs) are incredibly compact but require thousands of sequential network evaluations per ray, leading to hours or days of training. On the other end, explicit 3D voxel grids allow near-instantaneous lookups but suffer from cubic memory scaling ($O(N^3)$). To double the resolution of a voxel grid, you must octuple its storage requirement.
Tensor factorization offers an elegant escape from this dilemma. Instead of storing a dense 3D grid or relying entirely on a slow MLP, we can decompose a high-dimensional scene tensor into low-rank, lower-dimensional components. This approach achieves the fast query speeds of grid-based methods while maintaining a memory footprint that scales sub-cubically.
The Mechanics of Decomposition
How do we break down a 3D volume? The most prominent strategy is Vector-Matrix (VM) decomposition, popularized in this domain by Anpei Chen et al. in TensoRF: Tensorial Radiance Fields.
Instead of representing a 3D grid of size $X \times Y \times Z$ directly, VM decomposition factorizes the volume into combinations of 1D vectors and 2D matrices. For instance, a feature at coordinate $(x, y, z)$ can be computed by multiplying a 1D vector along the $Z$-axis with a 2D matrix spanning the $XY$-plane:
$$A(x,y,z) = \sum_{r} v_z^{(r)}(z) \cdot M_{xy}^{(r)}(x,y)$$
By summing a small number of these low-rank outer products across all three axis-plane combinations ($XY \times Z$, $XZ \times Y$, $YZ \times X$), we can reconstruct complex 3D structures.
The memory savings are drastic. A dense grid requires $O(N^3)$ parameters. VM decomposition requires only $O(N^2 + N)$ parameters per rank. As shown by Shuai Guo et al. in Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views, this extreme parameter compression allows for high-fidelity reconstructions with a fraction of the memory footprint.
Alternatively, Canonical Polyadic (CP) decomposition pushes this further by factorizing the 3D volume entirely into 1D vectors along the primary axes ($X$, $Y$, and $Z$), reducing the memory scaling to a purely linear $O(N)$. Junjin Xiao leverages this in NECA: Neural Customizable Human Avatar to compress large 3D feature grids into highly compact 1D vectors, avoiding the prohibitive computational costs of dense volumetric grids.
Why It Works: Exploiting Structural Redundancy
Why can a complex 3D scene be compressed into a few vectors and matrices without losing sharp details? The answer lies in the structural redundancy of the physical world.
Most 3D scenes are not chaotic noise; they consist of continuous surfaces, smooth lighting gradients, and empty space. In a dense voxel grid, empty space consumes just as much memory as a highly detailed surface. Tensor decomposition naturally exploits this sparsity. A flat wall or a uniform sky can be perfectly represented by a low-rank matrix and vector product, leaving the model's capacity to focus on regions with high geometric or textural complexity.
Furthermore, VM decomposition strikes a unique balance compared to alternative representations like tri-planes (which use three 2D planes). As noted by Lourdes Agapito et al. in NViST: In the Wild New View Synthesis from a Single Image with Transformers, VM decomposition is exceptionally effective at modeling general, non-object-centric environments because it preserves complex spatial relationships more efficiently than tri-planes. Similarly, Qi Ye et al. observe in LDM: Large Tensorial SDF Model for Textured Mesh Generation that tensorial representations capture high-frequency geometric details without the high-frequency noise often introduced by tri-plane projections.
Generalizing Beyond Radiance Fields
The power of tensorial decomposition extends far beyond standard color and density fields. Because it acts as a general, memory-efficient mathematical backbone for volumetric data, researchers have adapted it to various facets of computer vision and graphics:
- Geometry and Signed Distance Fields (SDFs): Instead of storing density, tensorial grids can store distance to the nearest surface. Jia Li et al. demonstrate in TensoSDF: Roughness-aware Tensorial Representation for Robust Geometry and Material Reconstruction that replacing large neural networks with VM factorization for geometry encoding cuts training time by 50% and inference by 70% while improving the capture of sharp edges.
- Inverse Rendering and Illumination: Decomposing a scene into its constituent physical properties (materials, lighting, and shape) is notoriously expensive. Haian Jin et al. utilize VM decomposition in TensoIR: Tensorial Inverse Rendering to drastically accelerate the query speeds required to simulate complex effects like shadows and indirect lighting. For real-time applications, Tat-Jen Cham et al. show in 3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting that representing incoming light as a continuous field via VM decomposition slashes the memory footprint of volumetric light data by up to 87.5%, enabling complex specular reflections in real-time.
- Extreme Compression and Portability: By combining CP-decomposition with efficient indexing, the storage requirements of neural scenes can plummet to the size of a standard web image. Jae Yong Lee et al. demonstrate in Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB that a 3D feature volume can be compressed down to ~150 KB, enabling high-quality 3D content delivery over standard web protocols. Similarly, Joo Chan Lee et al. apply tensor factorization to both spatial coordinates and Gaussian attributes in F-3DGS: Factorized Coordinates and Representations for 3D Gaussian Splatting, reducing the storage footprint of 3D Gaussian Splatting by over 90%.
- Dynamic and Articulated Avatars: For modeling humans and faces, tensor-based canonical representations provide the speed and detail necessary to handle complex deformations. Chuhan Chen utilizes a tensor-based decomposition in Implicit Neural Head Synthesis via Controllable Local Deformation Fields to capture fine-grained facial details like wrinkles while maintaining a manageable computational footprint.
By moving away from the rigid cubic scaling of dense grids and the heavy computational tax of coordinate-based MLPs, low-rank tensor decomposition provides a highly flexible, fast, and compact framework for 3D vision. It proves that the complex, high-dimensional visual world can be remarkably well-approximated by the simple interplay of lines and planes.
Key Foundational Work
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021) — contributed 1 concept elements to this direction.
WebGL-based streaming and presentation of objects with bidirectional texture functions (2013) — contributed 1 concept elements to this direction.