To render a single pixel in a standard Neural Radiance Field (NeRF) or volumetric representation, a ray is cast through a 3D scene, and a neural network is queried at dozens or hundreds of sample points along that ray to integrate color and density.
Mathematically, this creates a strict computational tax: the rendering cost scales linearly with the number of samples per ray, and quadratically with the image resolution. If you scale an output from a modest $256 \times 256$ resolution to a high-fidelity $1024 \times 1024$ resolution, the number of pixels increases sixteenfold. With 64 samples per ray, a single image requires over 67 million neural network evaluations. As noted by Jiatao Gu et al. in StyleNeRF: A Style-based 3D Aware Generator for High-resolution Image Synthesis, this poor scalability has historically restricted 3D-aware generative models to low-resolution outputs, preventing them from matching the visual quality of pure 2D architectures.
To bypass this "volumetric memory wall," researchers have developed a spectrum of strategies that decouple the geometric representation from the final high-resolution pixel grid.
The Resolution versus Consistency Dilemma
Because dense volumetric sampling is so computationally prohibitive, early 3D generative frameworks faced a difficult trade-off. As Andreas Geiger et al. discuss in GIRAFFE: Representing Scenes As Compositional Generative Neural Feature Fields, generating high-resolution images using pure 3D representations is often computationally prohibitive due to the intensive sampling required for projection.
To circumvent this, many early architectures resorted to a hybrid approach: volume-rendering a low-resolution feature map or image, and then using a 2D convolutional neural network (such as a super-resolution upsampler) to scale it up to the target high resolution. However, this introduces a severe geometric flaw. Because the 2D upsampler operates entirely on the flat image plane without explicit knowledge of the underlying 3D geometry, features tend to "stick" to the screen or warp unnaturally when the camera moves. This breaks multi-view consistency and degrades the underlying 3D structure.
This challenge is echoed across diverse generative domains. For instance, in focus-based 3D synthesis, Takuhiro Kaneko notes in Unsupervised Learning of Depth and Depth-of-Field Effect From Natural Images With Aperture Rendering Generative Adversarial Networks that the computational overhead of differentiable rendering layers restricts focus-based 3D generative models to low-resolution image synthesis.
Factorized and Hybrid Representations
One intuitive way to scale volumetric rendering is to change how the 3D space is stored. Instead of querying a dense 3D voxel grid or evaluating an un-factorized implicit function for every point in space, the scene can be factorized into lower-dimensional projections, such as orthogonal planes.
Standard Volumetric NeRF:
[Ray] ---> [Sample Point (x,y,z)] ---> [Deep MLP Evaluation] ---> (Color, Density)
*Repeated 64+ times per pixel*
Factorized Plane Representation (e.g., Tri-Planes / OrthoPlanes):
/ [XY Plane Feature]
[Point] --+--> [YZ Plane Feature] ---> [Bilinear Interpolation] ---> [Lightweight Decoder]
\ [XZ Plane Feature]
*Drastically reduces network depth and memory overhead*
By projecting 3D features onto 2D planes (such as Tri-Planes or Multiplane Images), the model can replace expensive multi-layer perceptron (MLP) evaluations with fast 2D bilinear lookups.
In Fused-Planes: Why Train a Thousand Tri-Planes When You Can Share?, Karim Kassab et al. address the computational bottleneck of volume rendering directly into high-resolution pixel space by sharing base planes across similar objects, drastically lowering the hardware requirements for large-scale 3D reconstruction.
Similarly, Honglin He et al. introduce OrthoPlanes: A Novel Representation for Better 3D-Awareness of GANs, which uses groups of parallel planes to scale 3D-aware generators to native high resolutions. This approach mitigates the extreme computational and memory overhead that usually limits direct high-resolution RGB synthesis.
For real-time applications, the authors of Generative Multiplane Neural Radiance for 3D-Aware Image Generation combine Multiplane Images (MPI) with an alpha-guided sampling strategy, achieving $1024 \times 1024$ resolution while bypassing the high memory consumption typical of pure implicit neural representations.
Shifting Rendering to Latent Spaces
Rather than rendering in the high-resolution pixel space, another highly effective paradigm shifts the volumetric integration process into a compressed latent space or onto specific manifolds.
Antoine Schnepf et al. demonstrate this concept in Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder. By training NeRFs directly on compressed image representations within a 3D-aware latent space, they bypass the computational bottlenecks of high-resolution image space, achieving massive speedups in training and rendering.
Alternatively, instead of compressing the spatial dimensions, models can restrict where the sampling occurs. Jianfeng Xiang et al. present GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds, which performs super-resolution directly on 2D "radiance manifolds" rather than in image space. This allows the network to leverage the efficiency of 2D convolutions while maintaining the strict structural integrity of a true 3D volume rendering process.
Intelligent and Adaptive Sampling
When a model must render in native high-resolution space, uniform sampling along camera rays becomes untenable. Instead, architectures must dynamically allocate computational resources to the regions of the scene that matter most.
Ivan Skorokhodov addresses this in EpiGRAF: Rethinking training of 3D GANs by combining patch-wise training with a scale-aware discriminator and a novel "Beta-distributed" sampling strategy. This allows the model to avoid the memory constraints of full-image rendering during training, eliminating the need for problematic 2D upsamplers.
Taking this further, Alex Trevithick et al. introduce a framework in What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs that abandons 2D super-resolution entirely. By utilizing a learned, adaptive sampler alongside a Signed Distance Function (SDF) representation, they scale neural volume rendering to native high resolutions. The adaptive sampler ensures that network evaluations are concentrated precisely around the scene's surfaces, allowing the model to evaluate millions of points along camera rays efficiently without sacrificing the fidelity of the underlying 3D representation.
The Volumetric Bottleneck in Fully 3D Data
The memory wall becomes even steeper when moving from 2D images of 3D scenes to generating fully 3D volumetric data, such as medical scans or articulated human bodies, where the entire 3D grid must be synthesized and preserved.
In the medical domain, Lennart Alexander Van der Goten et al. note in Conditional De-Identification of 3D Magnetic Resonance Images that generating high-resolution 3D volumetric data is constrained by massive memory requirements and computational costs compared to 2D image synthesis, hindering the creation of high-fidelity anatomical structures.
To manage these demands during full volumetric diffusion, George Webber et al. argue in Likelihood-Scheduled Score-Based Generative Modeling for Fully 3D PET Image Reconstruction that scaling generative models to high-resolution 3D volumes requires moving toward more efficient architectures like 2.5D representations or latent diffusion models.
Following this trajectory, Hyungjin Chung et al. propose Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models. By modeling a 3D volume as the product of distributions from two perpendicular planes, they capture global 3D dependencies without the prohibitive video memory requirements of native 3D diffusion models.
Native 3D Volumetric Diffusion:
[3D Voxel Grid] ---> Massive VRAM & Compute Wall
Perpendicular 2D Decomposition (TPDM):
[Axial Plane Diffusion] x [Coronal Plane Diffusion] ---> Reconstructed 3D Volume
For complex, articulated subjects like full-body humans, Zhuoqian Yang et al. demonstrate in 3DHumanGAN: 3D-Aware Human Image Generation with 3D Pose Mapping that modulating a pixel-wise independent 2D convolutional backbone with a 3D pose mapping network successfully overcomes the memory overhead that typically restricts purely volumetric rendering techniques when scaling to high resolutions.
Key Foundational Work
GIRAFFE: Representing Scenes As Compositional Generative Neural Feature Fields (2021) — contributed 1 concept elements to this direction.
Conditional De-Identification of 3D Magnetic Resonance Images (2021) — contributed 1 concept elements to this direction.
Synthesize Then Compare: Detecting Failures and Anomalies for Semantic Segmentation (2020) — contributed 1 concept elements to this direction.