Hybrid Scene Representations for Realism and Physics

97 concepts72 papers2 proposals

Computer graphics and robotic simulation have long faced a fundamental tension between how a scene looks and how it behaves.

On one hand, implicit neural representations—such as Neural Radiance Fields (NeRFs) and coordinate-based Signed Distance Fields (SDFs)—excel at capturing photorealistic visual details and complex, non-manifold topologies. However, because they encode geometry as continuous functions within a neural network's weights, they lack a discrete surface. This makes it computationally prohibitive for a physics engine to calculate a collision or for a robot to plan a manipulation path.

On the other hand, explicit representations—such as polygonal meshes, point clouds, and volumetric grids—are the native language of physics engines and spatial planners. They allow for fast ray-casting and rigid-body dynamics, but they struggle to represent fine-grained, view-dependent textures, complex boundaries, or soft geometries like hair and smoke.

To resolve this tension, a powerful paradigm has emerged: hybrid explicit-implicit representations. By strategically combining discrete geometric substrates with continuous neural fields, these architectures achieve the best of both worlds—photorealistic visual fidelity and real-time physical interactivity.


Decoupling Appearance from Physics

The most direct way to build a hybrid environment is to separate the visual rendering layer from the physical interaction layer. Instead of forcing a single data structure to handle both photons and contact forces, these frameworks maintain two synchronized representations of the same scene.

┌─────────────────────────────────────────────────────────┐
│                     Hybrid Scene                        │
└────────────────────────────┬────────────────────────────┘
                             │
              ┌──────────────┴──────────────┐
              ▼                             ▼
   ┌────────────────────┐        ┌────────────────────┐
   │    Visual Layer    │        │   Physics Layer    │
   │ (Gaussian Splats)  │        │  (Invisible Mesh)  │
   └──────────┬─────────┘        └──────────┬─────────┘
              ▼                             ▼
     Photorealistic Views          Collision Boundaries

For instance, in robotic navigation and manipulation, an agent needs to see a photorealistic world but collide with solid boundaries. In Vid2Sim, Zhizheng Liu et al. pair a 3D Gaussian Splatting (3DGS) visual layer with a hidden, geometry-consistent mesh layer. The 3DGS layer handles high-fidelity rendering, while the underlying mesh computes rigid-body collisions. This decoupling allows reinforcement learning agents to train in a "digital twin" of the real world that respects both visual realism and physical constraints.

A similar strategy is employed by Leonardo Barcellona et al. in Dream to Manipulate, where object-centric visual assets captured via 3DGS are converted into geometric meshes compatible with physics simulators. By separating these roles, frameworks like AutoVFX, developed by Hao-Yu Hsu et al. in AutoVFX: Physically Realistic Video Editing from Natural Language Instructions, can simulate complex physical interactions—such as inserting a synthetic rigid object into a real-world video—by calculating shadows and collisions on a proxy mesh while rendering the final output through a high-fidelity radiance field.


Anchoring Neural Fields to Geometric Skeletons

While decoupling works well for rigid environments, dynamic scenes—such as animating a human face or manipulating a deformable object—require the visual and physical layers to deform tightly together. To achieve this, researchers anchor continuous neural fields directly onto explicit, moving geometric skeletons.

In this setup, a coarse explicit model (like a parametric mesh or a set of sparse primitives) provides structural stability and global tracking. Meanwhile, an implicit neural field is bound to the components of this mesh to handle high-frequency details that the mesh cannot explicitly parameterize.

  Explicit Mesh Vertex          Local Neural Field
     (Tracked Base)            (Fills in Details)
         ───────┐                 ┌───────────┐
                │                 │  Wrinkles │
                ▼                 │  Hair     │
                ● ──────────────► │  Glasses  │
                                  └───────────┘

For example, in DRAWER, Hongchi Xia et al. anchor 3D Gaussian visual primitives directly to the faces of an extracted Neural Signed Distance Field mesh. When an object like a drawer is pulled open, the visual primitives move in perfect spatial alignment with the physical collision boundaries.

This anchoring mechanism is also highly effective for modeling human avatars. Traditional 3D Morphable Models (3DMMs) track facial expressions well but fail to capture non-manifold structures like hair, glasses, or the inside of the mouth. To fix this, Jingxiang Sun et al. introduced Next3D, which projects neural textures from a coarse 3D mesh into orthogonal feature planes via rasterization. This hybrid approach allows the model to inherit the precise deformation control of a mesh while maintaining the topological flexibility needed to render complex hairstyles and accessories.

Similarly, Boyi Jiang et al. present SelfRecon, combining an explicit mesh template with an implicit SDF in a shared canonical space to preserve global shape consistency while capturing fine-grained clothing wrinkles.


Hybrid Encodings for Real-Time Mapping

Beyond graphics and simulation, hybrid representations solve a critical scalability problem in Simultaneous Localization and Mapping (SLAM). Purely coordinate-based neural networks (like standard MLPs) provide excellent global smoothness and can fill in unobserved "holes" in a scene due to their inductive biases. However, they suffer from catastrophic forgetting and are slow to optimize in real time.

Hybrid neural encodings overcome this by combining discrete spatial data structures—such as multi-resolution hash-grids or tri-planes—with compact coordinate networks.

In Co-SLAM, Hengyi Wang et al. demonstrate that combining multi-resolution hash-grids with coordinate-based "one-blob" encodings allows a system to simultaneously capture high-frequency textures and maintain global surface smoothness. The discrete grid ensures rapid convergence and localized updates, while the neural network interpolates between grid points to ensure structural continuity.

To scale this to multi-room environments, Tianchen Deng et al. developed PLGSLAM, which uses a progressive local-to-global scene representation. By storing feature vectors in localized 2D grids and processing them through a shared network, the system bounds memory overhead to the surface area of the scene rather than the total volume, enabling real-time, drift-free tracking and dense reconstruction.


Why the Hybrid Approach Works

The success of hybrid explicit-implicit representations comes down to a core principle of representation alignment: allocating computational primitives to the mathematical operations they suit best.

  1. Photons vs. Forces: Rendering view-dependent effects requires evaluating continuous volume integrals or blending overlapping primitives (like Gaussians). Physics engines require discrete boundary intersections. Factorizing the scene into an implicit rendering field and an explicit geometric substrate allows both pipelines to run natively without translation bottlenecks.
  2. Global vs. Local Dynamics: Tracking a moving object or an articulating joint is a low-frequency, global operation best handled by explicit skeletons (meshes or sparse graphs). Modeling the subtle surface deformations, lighting changes, or fine details is a high-frequency, local operation best handled by neural fields or feature grids anchored to that skeleton.
  3. Memory vs. Continuity: Dense 3D voxel grids scale poorly with volume ($O(N^3)$), while pure MLPs are slow to train. Hybrid encodings store sparse features in explicit 2D/3D structures (scaling with surface area $O(N^2)$) and use small, efficient MLPs to decode continuous coordinates, achieving high-frequency detail alongside global consistency.

By moving away from monolithic scene representations, these hybrid approaches provide the foundational architecture needed for truly interactive, photorealistic digital twins.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_explicit_implicit_representation_fd553ca8,
  title = {hybrid explicit-implicit representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-explicit-implicit-representation/rdr_fd553ca839124e91b1cd081a9580d456},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_explicit_implicit_representation_fd553ca8_v1,
  title = {hybrid explicit-implicit representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-explicit-implicit-representation/rdr_fd553ca839124e91b1cd081a9580d456/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake