Disentangling Geometry and Appearance in 3D Scene Stylization

16 concepts11 papers1 proposal

Imagine trying to sculpt a physical statue of a lion and then paint it in the expressive, swirling style of Vincent van Gogh’s The Starry Night. If you are not careful, the heavy, high-frequency brushstrokes of the paint might distort the physical form of the lion, making its sharp jawline look bumpy or causing its mane to lose its structural definition.

In the digital world of 3D scene editing and stylization, neural networks face an identical challenge. When we ask an optimization algorithm to apply an artistic style or a text prompt to a 3D representation—such as a Neural Radiance Field (NeRF) or a point cloud—the network encounters a fundamental optimization shortcut: it is often mathematically easier to mimic complex 2D textures by physically warping the 3D geometry rather than cleanly altering the surface color.

This phenomenon is known as geometry-appearance entanglement. Without explicit intervention, style transfer algorithms routinely introduce floating artifacts, blurred object boundaries, or catastrophic shape collapse. To solve this, a prominent research direction focuses on cleanly decoupling a scene's underlying physical structure from its visual appearance.

The Core Problem: Geometry as an Optimization Crutch

Why do standard neural networks conflate shape and style? As explained by Peng Wang et al., when a model attempts to minimize the difference between a 3D scene's appearance and a 2D style reference, it treats all trainable parameters as fair game. If the network's color-generating components struggle to represent the intricate, high-frequency details of an artistic style, the network will begin altering the density or spatial coordinates of the scene instead. It effectively carves the brushstrokes directly into the 3D skeleton.

The consequences of this entanglement are severe across different 3D representations:

  • Shape Deformation: As observed by Ipek Oztas et al., injecting style features without proper disentanglement can cause the entire 3D shape to deform or collapse entirely to match the patterns of the style reference.
  • Boundary Blurring: Jiafu Chen et al. note that joint optimization approaches frequently lead to a trade-off where complex artistic textures result in blurred object boundaries and unstable physical shapes.
  • Global Distortion: When editing specific regions or objects, entanglement causes changes to bleed outward. Moneish Kumar et al. point out that separating target style from the scene's physical representation is necessary to prevent localized edits from causing global geometric distortions.

To overcome these limitations, researchers have developed two primary strategies: explicit architectural freezing and coordinate-space decoupling.

Strategy 1: Architectural Decoupling and Parameter Freezing

The most direct way to prevent an optimization process from ruining a scene's geometry is to make the geometry mathematically unchangeable.

In implicit neural representations like NeRFs, a scene is typically parameterized by two distinct outputs: volume density (which dictates the 3D structure) and color/radiance (which dictates appearance). Hong-Wing Pang demonstrates an elegant framework that explicitly splits the neural representation into separate branches for density and color. By freezing the density branch entirely and optimizing only the appearance branch for the new style, the underlying 3D shape is guaranteed to remain intact.

[Input 3D Coordinates]
          │
          ├──► [Density Branch] ───► (FREEZING preserves structural geometry)
          │
          └──► [Appearance Branch] ──► [Style Optimization] ──► (Alters color only)

This parameter-freezing strategy serves as the foundation for several distinct applications:

  • Text-Driven Stylization: The TeSTNeRF framework adopts this approach by freezing pre-trained density parameters and optimizing only the radiance components. This ensures that abstract text prompts (e.g., "a cubist painting") modify only the visual texture, preventing structural distortions and eliminating the multi-view "flickering" common in unconstrained video stylization.
  • High-Fidelity Watermarking: Decoupling is equally valuable for security. Xingyu Zhu et al. demonstrate that by modifying only the color mapping of a neural representation while keeping its geometry fixed, they can embed high-accuracy data watermarks during the generation process without degrading visual quality or introducing structural artifacts.

Strategy 2: Mapping to Alternative Coordinate Spaces

While freezing parameters works well for static, pre-reconstructed scenes, it is less flexible when you need to perform simultaneous reconstruction and stylization, or when the edit requires a controlled blend of both geometric and stylistic changes.

To achieve disentanglement without completely locking the network, Jiafu Chen et al. introduce a method that projects 3D coordinates into a shared 2D "style pattern space" using a UV mapping network. By forcing the artistic style to express itself within this dedicated 2D coordinate space rather than the 3D volume, the stylization process is prevented from distorting the underlying 3D structure. This results in significantly sharper visual boundaries and more stable geometric representations.

For complex tasks like editing human faces, where an edit might require changing an expression (a geometric change) while applying a style (an appearance change), a deeper decomposition is required. Chunyi Sun et al. tackle this by mapping 3D representations to the latent spaces of 2D Generative Adversarial Networks (GANs). By using a differentiable latent code adjustor and a hidden mapper, they balance significant stylistic modifications with the strict preservation of the subject's original identity.

Generalizing Across Representations

The mandate to decouple shape from style holds true regardless of the underlying 3D data structure.

For explicit representations like point clouds, the authors of PSNet designed a network that independently processes geometric coordinates and color information. This architectural separation allows users to independently transfer the geometric structure of one asset and the color style of another.

Similarly, in the context of multi-view synthesis, Nail Ibrahimli et al. emphasize that balancing geometric preservation with style flexibility is the core hurdle to achieving rapid, high-quality results. By combining explicit multi-view stereo geometry learning with flexible style networks, and as corroborated by the hypernetwork approaches of Pei-Ze Chiang et al., decoupling ensures that artistic transformations do not degrade the underlying 3D reconstruction.

Ultimately, geometry-appearance disentanglement transforms 3D stylization from an unpredictable process prone to structural collapse into a precise, controllable tool. By removing the network's ability to use geometry as a shortcut for texture, researchers ensure that 3D assets retain their physical identity, no matter what style they are painted with.

Key Foundational Work

Stylizing 3D Scene via Implicit Representation and HyperNetwork (2022) — contributed 1 concept elements to this direction.
PSNet: A Style Transfer Network for Point Cloud Stylization on Geometry and Color (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometry_appearance_disentanglement_56e1ad2c,
  title = {geometry-appearance disentanglement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometry-appearance-disentanglement/rdr_56e1ad2cd8a44326bf269d1fea7f80b5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometry_appearance_disentanglement_56e1ad2c_v1,
  title = {geometry-appearance disentanglement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometry-appearance-disentanglement/rdr_56e1ad2cd8a44326bf269d1fea7f80b5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake