Baking Neural Radiance Fields into Textured Meshes

61 concepts49 papers1 proposal

Neural Radiance Fields (NeRFs) revolutionized 3D scene reconstruction by representing geometry and view-dependent appearance as continuous volumetric functions. However, rendering these representations requires volumetric ray marching: casting a ray for every pixel and querying a deep multi-layer perceptron (MLP) hundreds of times along that ray to sample density and color. This process is computationally devastating, making real-time deployment on consumer hardware or web browsers exceptionally difficult.

The fundamental issue is an architectural mismatch. Modern graphics processing units (GPUs) are not built for continuous volumetric sampling; they are highly optimized for the polygon rasterization pipeline, which projects discrete triangles onto a screen and processes them with massive pixel-level parallelism. To bridge this gap, a major research direction has emerged: distilling or "baking" the continuous knowledge of implicit neural fields into explicit graphics primitives—such as triangle meshes and standard texture maps—that can be rendered using traditional hardware-accelerated rasterization.

Decoupling Geometry from Appearance

The core insight behind this hybrid approach is that we can decouple the task of finding where a surface is (geometry) from what it looks like from a specific angle (view-dependent appearance). Instead of evaluating an MLP at hundreds of empty or dense points along a ray, we can extract an explicit geometric proxy of the scene.

Once this proxy geometry is established, the rendering pipeline shifts from ray marching to standard polygon rasterization. The GPU's native hardware instantly resolves visibility and determines exactly which triangle covers a given pixel. The computational burden is thus shifted from the depth of the volume to the number of visible triangles, unlocking massive performance gains.

Extracting and Anchoring Explicit Geometry

To convert a "foggy" neural volume into a clean geometric surface, researchers have proposed several strategies to extract or anchor geometry from implicit fields.

One approach is to train a neural field using a Signed Distance Function (SDF) or density field, and then extract a static triangle mesh using isosurface extraction techniques like Marching Cubes. For instance, Lior Yariv et al. extract a high-quality triangle mesh from a trained neural SDF in BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis, allowing the asset to be integrated directly into traditional graphics engines for physics simulations and collision detection. Similarly, Fabian Manhardt et al. use a Signed Surface Approximation Network in NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes to interpret a NeRF's volumetric density as a distinct surface mesh, enabling real-time rendering and physics-based interactions.

Instead of extracting a mesh post-training, other methods anchor the neural representation to explicit geometric structures from the start. Jonas Kulhanek et al. represent radiance fields using a tetrahedral mesh generated via Delaunay triangulation in Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra, storing neural features at the vertices and using barycentric interpolation for fast ray-triangle intersection queries. For dynamic objects like human bodies, Jing Wen et al. anchor Gaussian primitives directly onto a standard deformable mesh template in GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh, combining the photorealism of Gaussian splatting with the structural utility of traditional animation pipelines.

Baking Neural Features into Texture Maps

Once the geometry is defined, the next challenge is representing complex, view-dependent appearance without relying on a heavy neural network at runtime. The solution is to "bake" the neural properties into standard 2D texture maps or vertex attributes.

To do this, the continuous surface of the extracted mesh must be mapped to a 2D coordinate space. Pratul P. Srinivasan et al. developed a neural field-based approach to UV mapping in Nuvo: Neural UV Mapping for Unruly 3D Representations that optimizes continuous functions via MLPs, producing high-quality texture atlases even for noisy or fragmented geometry.

With a UV mapping in place, high-dimensional neural feature vectors or view-dependent coefficients (such as Spherical Gaussians or Spherical Harmonics) can be stored directly in standard texture images. Sara Rojas et al. demonstrate in Re-ReND: Real-time Rendering of NeRFs across Devices that baking pre-computed neural features into standard texture maps allows the rendering process to be offloaded entirely to hardware-accelerated lookups. In another variation, Christian Richardt et al. store learnable feature vectors directly on mesh vertices in Learning Neural Duplex Radiance Fields for Real-Time View Synthesis, projecting this high-dimensional information into screen-space buffers via standard hardware rasterization.

Hardware-Accelerated Shading and Cross-Platform Deployment

The final rendering step completely bypasses specialized machine learning frameworks like CUDA. When the mesh is rasterized, the GPU interpolates the baked features across the surface of the visible triangles in screen space. A lightweight fragment shader or a tiny deferred MLP then reads these interpolated features and computes the final view-dependent color.

By replacing volumetric ray marching with standard polygon rasterization and texture lookups, these hybrid architectures achieve dramatic speedups. Jeffrey Yunfan Liu et al. demonstrate in Real-Time Neural Rasterization for Large Scenes that combining a scaffold mesh with neural textures and lightweight MLP shaders can render city-scale scenes at over 100 FPS, outperforming traditional volumetric methods by up to 80x. For massive environments, Kaiwen Song et al. combine baked density and color textures with a small deferred MLP and a dynamic Level-of-Detail (LOD) system in City-on-Web: Real-Time Neural Rendering of Large-Scale Scenes on the Web to deliver real-time 1080p rendering on mid-range consumer GPUs.

Because the final assets are stored as standard geometric and texture formats (such as OBJ and PNG), they can be deployed natively across diverse platforms. The authors of MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures show that converting implicit representations into explicit textured polygon meshes enables interactive neural rendering on common mobile devices and web browsers via WebGL, utilizing the native z-buffering and rasterization hardware built into every modern device. Similarly, Jiaming Gu demonstrates in UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene that converting neural radiance fields into textured meshes with varying levels of detail allows for real-time, 4K interactive rendering directly within commercial game engines like Unreal Engine 4.

Why It Works: Shifting the Computational Burden

To understand why this paradigm is so effective, consider the computational complexity shift. Traditional volumetric neural rendering scales with both the image resolution and the number of samples along each ray:

$$\text{Cost} \propto \text{Pixels} \times \text{Samples per Ray} \times \text{MLP Complexity}$$

By contrast, the hybrid mesh-neural rasterization pipeline decouples these terms. The hardware rasterizer determines surface intersections instantly via built-in z-buffering. The neural network or fragment shader is evaluated exactly once per visible pixel, and its mathematical complexity is drastically reduced because it only needs to model view-dependent shading variations, not the underlying 3D geometry or density:

$$\text{Cost} \propto (\text{Triangles} \times \text{Rasterization Cost}) + (\text{Pixels} \times \text{Lightweight Shader Cost})$$

By prioritizing fast memory lookups over heavy coordinate-based neural network evaluations, this research direction successfully bridges the gap between the photorealistic fidelity of neural fields and the extreme efficiency of traditional graphics pipelines.

Key Foundational Work

Neural Lumigraph Rendering (2021) — contributed 2 concept elements to this direction.
Mixture of Volumetric Primitives for Efficient Neural Rendering (2021) — contributed 1 concept elements to this direction.
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_mesh_neural_rendering_576e7874,
  title = {hybrid mesh-neural rendering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-mesh-neural-rendering/rdr_576e787408b44973944b673d82e3651e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_mesh_neural_rendering_576e7874_v1,
  title = {hybrid mesh-neural rendering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-mesh-neural-rendering/rdr_576e787408b44973944b673d82e3651e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake