NeRFlex: Resource-aware Real-time High-quality Rendering of Complex Scenes on Mobile Devices

Zhe Wang, Yifei Zhu

Abstract

Neural Radiance Fields (NeRFs) have transformed 3D reconstruction but remain computationally prohibitive for mobile platforms. NeRFlex introduces a resource-aware framework that enables high-quality, real-time NeRF rendering on mobile devices by decomposing complex scenes into optimized sub-networks. By utilizing a dynamic programming approach to solve the resource allocation problem, NeRFlex achieves smooth performance (up to 35 FPS) while maintaining strict adherence to mobile memory constraints. Our evaluation shows that NeRFlex outperforms existing baselines in visual quality and efficiency, particularly in complex, multi-object environments.

Introduction

While Neural Radiance Fields (NeRFs) have revolutionized the ability to synthesize photorealistic novel views of 3D scenes, their deployment on mobile devices remains a significant challenge. The primary bottleneck is a fundamental trade-off: high-quality rendering typically requires massive memory and computational overhead, while simplified models often lose the high-frequency details necessary for complex scenes.

Existing "baked" NeRF solutions, which pre-compute data into meshes or voxels, improve rendering speed but frequently ignore the strict memory limits of mobile browsers and hardware. Furthermore, single-network architectures often struggle to capture the nuances of scenes containing multiple distinct objects. NeRFlex addresses these limitations through a "divide-and-conquer" pipeline that optimizes resource distribution across a scene.

Methodology

NeRFlex employs a three-stage pipeline designed to maximize visual fidelity within a fixed resource budget.

1. Detail-Oriented Segmentation

The system first performs a spatial decomposition of the scene. Using object detection and frequency analysis, NeRFlex identifies "complex" regions—areas with high-frequency textures or intricate geometries. These regions are assigned dedicated sub-networks, while simpler, low-frequency regions are grouped. This ensures that the model's learning capacity is concentrated where it provides the most visual impact.

2. Lightweight White-Box Profiler

To avoid the overhead of black-box AI performance predictors, NeRFlex utilizes a mathematical "white-box" profiler. This model maps two primary parameters—mesh granularity ($g$) and texture patch size ($p$)—to predicted visual quality (SSIM) and file size. This allows the system to estimate the "cost" and "value" of various configurations analytically before the training phase begins.

3. DP-based Configuration Selector

NeRFlex treats the optimization of these sub-networks as a Multiple-Choice Knapsack Problem (MCKP). Given a global memory limit (e.g., 240MB for an iPhone), the system uses a dynamic programming (DP) algorithm to select the optimal configuration for each sub-network. By "stealing" memory from simple regions (like a flat floor) and reallocating it to complex objects (like a detailed centerpiece), the system maximizes the overall scene quality without exceeding the hardware's capacity.

Experimental Results

Performance and Efficiency

NeRFlex was evaluated on standard synthetic 360-degree benchmarks and real-world datasets (LLFF). On an iPhone 13, the system achieved an average of 35 FPS, while maintaining 25 FPS on a Pixel 4. Crucially, NeRFlex successfully rendered complex scenes within a 240MB memory envelope. In comparison, baselines such as Block-NeRF required over 600MB for similar scenes, frequently causing mobile WebGL engines to crash.

Visual Quality

In complex scenarios, NeRFlex achieved SSIM scores ranging from 0.88 to 0.94. This significantly outperforms single-network baselines like MobileNeRF and Mip-NeRF 360, which often struggle to exceed an SSIM of 0.84 in multi-object environments. The pre-rendering optimization process, including segmentation and configuration selection, is highly efficient, taking approximately 5.9 seconds on a cloud server.

Discussion and Limitations

One of the key insights of NeRFlex is the use of maximum frequency across viewpoints rather than average frequency to guide segmentation. This approach aligns with human visual perception, where blurriness from a single angle can break the illusion of realism.

However, limitations remain. While the rendering is real-time, the initial training of the sub-networks requires several hours on high-end desktop GPUs (e.g., RTX 3090). Additionally, the current framework is optimized for static scenes; extending this resource-aware allocation to dynamic environments with moving objects represents a significant area for future research. Finally, while the system optimizes baked data, further gains could potentially be found by exploring the quantization of the Multi-Layer Perceptron (MLP) weights themselves.

Conclusion

NeRFlex demonstrates that high-quality NeRF rendering is achievable on mobile devices through intelligent resource management. By shifting from a monolithic network approach to a segmented, optimized architecture, NeRFlex provides a practical path for bringing photorealistic 3D content to the mobile web and handheld AR applications.

Summary updated

Abstract

Neural Radiance Fields (NeRF) is a cutting-edge neural network-based technique for novel view synthesis in 3D reconstruction. However, its significant computational demands pose challenges for deployment on mobile devices. While mesh-based NeRF solutions have shown potential in achieving real-time rendering on mobile platforms, they often fail to deliver high-quality reconstructions when rendering practical complex scenes. Additionally, the non-negligible memory overhead caused by pre-computed intermediate results complicates their practical application. To overcome these challenges, we present NeRFlex, a resource-aware, high-resolution, real-time rendering framework for complex scenes on mobile devices. NeRFlex integrates mobile NeRF rendering with multi-NeRF representations that decompose a scene into multiple sub-scenes, each represented by an individual NeRF network. Crucially, NeRFlex considers both memory and computation constraints as first-class citizens and redesigns the reconstruction process accordingly. NeRFlex first designs a detail-oriented segmentation module to identify sub-scenes with high-frequency details. For each NeRF network, a lightweight profiler, built on domain knowledge, is used to accurately map configurations to visual quality and memory usage. Based on these insights and the resource constraints on mobile devices, NeRFlex presents a dynamic programming algorithm to efficiently determine configurations for all NeRF representations, despite the NP-hardness of the original decision problem. Extensive experiments on real-world datasets and mobile devices demonstrate that NeRFlex achieves real-time, high-quality rendering on commercial mobile devices.
BibTeX
@inproceedings{wang2025nerflex,
  author = {Zhe Wang and Yifei Zhu},
  title = {{NeRFlex: Resource-aware Real-time High-quality Rendering of Complex Scenes on Mobile Devices}},
  booktitle = {ICDCS},
  year = {2025},
  url = {https://openreview.net/forum?id=OFljVeqzjv},
}
Figure