Neural Proxies for Non-Differentiable Rendering

This direction is split into other directions: differentiable proxy rendering. Its last version stays readable and citable here.

Version 1 of 140 papers56 conceptsfrozen 2026-10-04

Deep learning thrives on gradients. To optimize a model, we need to know how a small change in an input parameter—like the curve of a 3D mesh or the roughness of a digital material—affects the final output image. However, most professional graphics engines, procedural generators, and physical simulators are "black boxes." They are non-differentiable, meaning they can produce an image from parameters, but they cannot tell you the mathematical gradient required to go backward from an image to those parameters.

To bridge this gap, researchers have developed a technique known as neural proxy rendering. Instead of trying to rewrite decades of graphics code to be differentiable, they train a neural network to "imitate" the renderer. This proxy acts as a differentiable surrogate, allowing gradients to flow through an otherwise impenetrable black box.

The Gradient Gap in Inverse Graphics

The goal of inverse graphics is to take a real-world observation (like a photo) and find the underlying parameters (shape, lighting, material) that would produce it. In Joint Material and Illumination Estimation from Photo Sets in the Wild, Tuanfeng Y. Wang and Tobias Ritschel demonstrate that while physically-based rendering is accurate, it is too slow for the inner loop of an optimization. By training a neural network to approximate the reflection operator and its gradients, they enable an optimizer to navigate the complex space of light and material efficiently.

This problem is even more acute when dealing with commercial software. In Face-to-Parameter Translation for Game Character Auto-Creation, researchers like Tianyang Shi and Changjie Fan faced a challenge: game engines use discrete, bone-driven parameters that are inherently non-differentiable. Their solution was to train a "neural imitator" that learns the mapping from character parameters to rendered pixels. Because the imitator is a neural network, it is fully differentiable, allowing the system to use standard backpropagation to "tune" a 3D avatar until it matches a user's selfie.

How Proxies Enable Optimization

A neural proxy works by creating a smooth, differentiable landscape over a discrete or jagged one. This enables several distinct research directions:

1. Automated Content Creation

Beyond faces, this approach extends to complex procedural systems. Yiwei Hu and colleagues in Node Graph Optimization Using Differentiable Proxies showed that non-differentiable "generator" nodes in material graphs (like those creating brick patterns) can be replaced by neural proxies. This allows an artist to provide a target photograph and have the system automatically optimize the scale, layout, and structural parameters of the procedural graph to match.

2. Physical-World Adversarial Attacks

Neural proxies are also a powerful tool for security research. To create an adversarial texture that fools a car's vision system, you need to know how that texture looks under different lighting and angles. Chaowei Xiao and Jia Deng used this in MeshAdv, employing a differentiable proxy renderer to propagate gradients from a 2D prediction back to 3D geometry. Similarly, DTA: Physical Camouflage Attacks using Differentiable Transformation Network by Naufal Suryanto uses a proxy to account for complex environmental factors like shadows and transparency provided by high-end game engines, ensuring the adversarial attack transfers from simulation to the real world.

3. Accelerating Complex Physics

Some rendering processes are technically differentiable but computationally ruinous. For example, Deep Polarization Cues for Single-Shot Shape and Subsurface Scattering Estimation deals with materials like jade or skin where light scatters internally. Instead of using a slow volumetric path tracer, Chenhao Li and team use a neural proxy to simulate the rendering, allowing for efficient image-space error backpropagation to refine material parameters.

Alternatives to Neural Proxies

While neural proxies are dominant, other researchers explore different ways to handle the "non-differentiable" barrier:

  • Softening Operations: In Dr.Bokeh, Yichen Sheng and colleagues replace discrete visibility tests (which are binary and thus have zero gradients) with "soft" continuous approximations using functions like tanh. This makes the renderer natively differentiable without needing a proxy network.
  • Reinforcement Learning: In Synthesizing Programs for Images using Reinforced Adversarial Learning, Yaroslav Ganin and Oriol Vinyals treat the renderer as a black-box environment. Instead of gradients, they use an RL agent (SPIRAL) that learns to "paint" by receiving rewards from a discriminator, bypassing the need for differentiability entirely.
  • Vector Primitives: Stroke-based Character Reconstruction by Zhewei Huang uses neural networks as proxies for drawing Bezier curves, filtering out noise by forcing the model to represent characters through geometric primitives rather than raw pixels.

Why It Works: The Surrogate Gradient

The intuition behind neural proxies is that the network doesn't need to be a perfect replica of the renderer; it only needs to be "locally correct" enough to provide a useful gradient direction. By approximating the behavior of the renderer, the proxy provides a surrogate gradient that points the optimizer toward the correct parameters. Once the optimization is complete, the final parameters can be plugged back into the high-fidelity, non-differentiable engine to produce the final, photorealistic result. This hybrid approach combines the mathematical convenience of deep learning with the established power of traditional graphics software.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_proxies_for_non_differentiable_re_9de0084b,
  title = {Neural Proxies for Non-Differentiable Rendering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-proxies-for-non-differentiable-rendering/rdr_9de0084bd299491e87f4ac9ea53b3b0a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_proxies_for_non_differentiable_re_9de0084b_v1,
  title = {Neural Proxies for Non-Differentiable Rendering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-proxies-for-non-differentiable-rendering/rdr_9de0084bd299491e87f4ac9ea53b3b0a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}