Coordinate-based neural representations, such as Neural Radiance Fields (NeRFs) and implicit occupancy networks, revolutionized 3D reconstruction by mapping continuous spatial coordinates directly to geometric and visual properties. However, early iterations of these models were fundamentally object-centric or restricted to bounded, room-sized environments. When tasked with reconstructing kilometer-scale urban landscapes or tracking long camera trajectories, they hit a strict computational and representational barrier: the capacity wall.
To scale neural fields to massive, unbounded environments, researchers had to move away from monolithic neural networks. The solution lies in spatial partitioning—breaking down global environments into localized, manageable, and hierarchical sub-units.
The Capacity Wall of Monolithic Networks
Why do standard neural fields fail when a scene grows too large? The problem stems from trying to compress an entire environment into the fixed parameter space of a single Multi-Layer Perceptron (MLP).
An MLP acts as a continuous function approximator. For a single object or a small room, the network has more than enough capacity to memorize the high-frequency details of geometry and view-dependent appearance. However, as the physical space expands to encompass entire streets or cities, the amount of geometric detail and visual complexity scales exponentially.
When forced to store this massive volume of data within a single set of fixed weights, the network suffers from severe optimization bottlenecks and catastrophic forgetting. Trying to optimize for a new street corner can inadvertently corrupt the weights representing a block a mile away. As demonstrated by Torsten Sattler et al. in Understanding the Limitations of CNN-Based Absolute Camera Pose Regression, encoding expansive environments into a single coordinate regression network causes significant performance degradation, often causing deep models to be outperformed by simple, non-parametric image retrieval baselines.
Furthermore, as noted by Zhenxing Mi et al. in Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields, a monolithic architecture requires an unsustainably large number of parameters to capture city-scale details, making training and rendering prohibitively slow and resource-intensive.
Slicing Up Space: Local Inductive Biases
To bypass the capacity wall, the core insight of modern spatial scalability research is to introduce a strong spatial inductive bias: divide and conquer. Instead of asking one global network to memorize an entire city, the environment is sliced into localized regions, each governed by its own independent set of parameters.
An early foundation for this approach was established by Songyou Peng et al. in Convolutional Occupancy Networks. The authors replaced global feature vectors with localized convolutional feature grids. By anchoring neural features to specific spatial coordinates, the representation ensures that the network only processes local information when reconstructing a given point in space, allowing implicit models to scale gracefully from isolated objects to multi-room scenes.
Monolithic Approach (Fails at Scale):
[ 3D Coordinate ] ---> [ Giant Global MLP ] ---> [ Color / Density ]
(Capacity Wall)
Spatially Partitioned Approach (Scales):
/---> [ Local Grid A ] ---\
[ 3D Coordinate ] ---> ----> [ Local Grid B ] ----> [ Small MLP ] ---> [ Color / Density ]
\---> [ Local Grid C ] ---/
This spatial partitioning yields three massive advantages:
- Constant-Time Inference: When rendering a specific ray or querying a local coordinate, the system only activates the parameters dedicated to that local sub-volume. The computational cost remains constant, regardless of whether the entire map is a single room or an entire continent.
- Prevention of Catastrophic Forgetting: Because updates are localized, optimizing the geometry of a newly observed building does not alter or degrade the representation of structures elsewhere in the scene.
- Parallelizable Training: Different regions of the environment can be trained simultaneously across multiple GPUs, eliminating the hardware bottlenecks of large-scale data ingestion.
Hierarchical Grids and Sparse Latent Codes
In practice, spatial partitioning is often implemented using hierarchical structures or sparse latent code grids. This ensures that memory is only allocated where surfaces actually exist, rather than wasting resources on empty space (such as the sky or air).
For real-time applications like Simultaneous Localization and Mapping (SLAM), Zihan Zhu et al. introduced a hierarchical, grid-based neural implicit representation in NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. By stratifying the environment into multi-level local feature grids (coarse, mid, and fine tracking levels), the system can reconstruct large, multi-room apartments in real time while maintaining a stable memory footprint.
Similarly, for wide-area camera localization, Shitao Tang et al. developed NeuMap: Neural Coordinate Mapping by Auto-Transdecoder for Camera Localization. Instead of relying on fragile global weights, the authors distributed scene information across a grid of sparse latent codes. A scene-agnostic transformer then decodes these local codes into 3D coordinates, providing exceptional scalability in expansive outdoor settings while drastically reducing storage requirements compared to traditional point clouds.
Dynamic Decomposition via Mixture of Experts
While rigid, hand-crafted grids work well for uniform environments, real-world scenes feature highly variable geometry and detail density. A wide-open square requires fewer parameters than an intricately detailed building facade.
To address this dynamically, Zhenxing Mi et al. proposed Switch-NeRF, which utilizes a Sparsely Gated Mixture of Experts (MoE) architecture for large-scale neural radiance fields. Instead of hardcoding spatial boundaries, the authors designed an end-to-end learnable gating network that automatically routes 3D coordinates to specialized subnetworks (the "experts").
[ 3D Coordinate ] ---> [ Learnable Gating Network ]
/ | \
v v v
[Expert 1] [Expert 2] [Expert 3]
(Facade) (Sky) (Ground)
This learnable decomposition allows different neural experts to specialize in distinct spatial regions or semantic components of a city. Because the gating mechanism is sparse, only a fraction of the total network parameters are executed for any single sample, maximizing representation capacity without inflating the computational budget during rendering.
Level of Detail and Graphics Integration
Slicing up neural scenes also unlocks a crucial bridge to classical computer graphics: Level of Detail (LoD) management. In traditional graphics, objects that are far from the camera are rendered using low-resolution meshes to save performance. Spatial partitioning allows neural fields to adopt the same principle.
In UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene, Jiaming Gu demonstrated that partitioning kilometer-scale environments into distinct sub-blocks and optimizing them into multi-resolution hierarchies enables interactive 4K rendering at high frame rates. When a user explores a massive virtual environment, the system dynamically streams and evaluates fine-grained local blocks near the camera, while distant backgrounds are handled by coarse, low-resolution representations.
By combining the continuous, photorealistic rendering of neural fields with the discrete, localized data structures of traditional graphics pipelines, spatial scalability transitions from a theoretical challenge to a practical reality for city-scale simulation, autonomous driving, and virtual exploration.
Key Foundational Work
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019) — contributed 2 concept elements to this direction.