Tensor Factorization for Efficient Volumetric Scene Representations

27 concepts20 papers3 proposals

When neural scene representations like Neural Radiance Fields (NeRF) emerged, they presented a stark trade-off between memory and speed. On one end, coordinate-based Multi-Layer Perceptrons (MLPs) are incredibly compact but require thousands of sequential network evaluations per ray, leading to hours or days of training. On the other end, explicit 3D voxel grids allow near-instantaneous lookups but suffer from cubic memory scaling ($O(N^3)$). To double the resolution of a voxel grid, you must octuple its storage requirement.

Tensor factorization offers an elegant escape from this dilemma. Instead of storing a dense 3D grid or relying entirely on a slow MLP, we can decompose a high-dimensional scene tensor into low-rank, lower-dimensional components. This approach achieves the fast query speeds of grid-based methods while maintaining a memory footprint that scales sub-cubically.

The Mechanics of Decomposition

How do we break down a 3D volume? The most prominent strategy is Vector-Matrix (VM) decomposition, popularized in this domain by Anpei Chen et al. in TensoRF: Tensorial Radiance Fields.

Instead of representing a 3D grid of size $X \times Y \times Z$ directly, VM decomposition factorizes the volume into combinations of 1D vectors and 2D matrices. For instance, a feature at coordinate $(x, y, z)$ can be computed by multiplying a 1D vector along the $Z$-axis with a 2D matrix spanning the $XY$-plane:

$$A(x,y,z) = \sum_{r} v_z^{(r)}(z) \cdot M_{xy}^{(r)}(x,y)$$

By summing a small number of these low-rank outer products across all three axis-plane combinations ($XY \times Z$, $XZ \times Y$, $YZ \times X$), we can reconstruct complex 3D structures.

The memory savings are drastic. A dense grid requires $O(N^3)$ parameters. VM decomposition requires only $O(N^2 + N)$ parameters per rank. As shown by Shuai Guo et al. in Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views, this extreme parameter compression allows for high-fidelity reconstructions with a fraction of the memory footprint.

Alternatively, Canonical Polyadic (CP) decomposition pushes this further by factorizing the 3D volume entirely into 1D vectors along the primary axes ($X$, $Y$, and $Z$), reducing the memory scaling to a purely linear $O(N)$. Junjin Xiao leverages this in NECA: Neural Customizable Human Avatar to compress large 3D feature grids into highly compact 1D vectors, avoiding the prohibitive computational costs of dense volumetric grids.

Why It Works: Exploiting Structural Redundancy

Why can a complex 3D scene be compressed into a few vectors and matrices without losing sharp details? The answer lies in the structural redundancy of the physical world.

Most 3D scenes are not chaotic noise; they consist of continuous surfaces, smooth lighting gradients, and empty space. In a dense voxel grid, empty space consumes just as much memory as a highly detailed surface. Tensor decomposition naturally exploits this sparsity. A flat wall or a uniform sky can be perfectly represented by a low-rank matrix and vector product, leaving the model's capacity to focus on regions with high geometric or textural complexity.

Furthermore, VM decomposition strikes a unique balance compared to alternative representations like tri-planes (which use three 2D planes). As noted by Lourdes Agapito et al. in NViST: In the Wild New View Synthesis from a Single Image with Transformers, VM decomposition is exceptionally effective at modeling general, non-object-centric environments because it preserves complex spatial relationships more efficiently than tri-planes. Similarly, Qi Ye et al. observe in LDM: Large Tensorial SDF Model for Textured Mesh Generation that tensorial representations capture high-frequency geometric details without the high-frequency noise often introduced by tri-plane projections.

Generalizing Beyond Radiance Fields

The power of tensorial decomposition extends far beyond standard color and density fields. Because it acts as a general, memory-efficient mathematical backbone for volumetric data, researchers have adapted it to various facets of computer vision and graphics:

By moving away from the rigid cubic scaling of dense grids and the heavy computational tax of coordinate-based MLPs, low-rank tensor decomposition provides a highly flexible, fast, and compact framework for 3D vision. It proves that the complex, high-dimensional visual world can be remarkably well-approximated by the simple interplay of lines and planes.

Key Foundational Work

FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021) — contributed 1 concept elements to this direction.
WebGL-based streaming and presentation of objects with bidirectional texture functions (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vector_matrix_volume_decomposition_2eb2bef5,
  title = {vector-matrix volume decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-matrix-volume-decomposition/rdr_2eb2bef5aab549db8e690956fff633a6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_vector_matrix_volume_decomposition_2eb2bef5_v1,
  title = {vector-matrix volume decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-matrix-volume-decomposition/rdr_2eb2bef5aab549db8e690956fff633a6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake