Engine-Verified Proxy Descent for Few-Target Inverse Rendering

Many graphics and physics engines cannot be differentiated: game character systems, path tracers for translucent materials, and procedural shaders contain branches and visibility tests that block gradients. The standard workaround is to train a neural network to imitate the engine, a differentiable proxy, and then recover the engine's input settings for a target image by running gradient descent through that imitation. Doing this well usually takes a lot of data. Shi et al. avoid slow per-image optimization by training a feed-forward "translator" on more than 200,000 CelebA faces (Neural Rendering for Game Character Auto-Creation). That route is closed when only a few dozen real target images exist, as is common in clinical and specialist settings (Towards AI-driven Next Generation Personalized Healthcare and Well-being). What remains is per-target optimization through the proxy. The same authors report that it works poorly: on LFW the earlier iterative method scores 0.6977 against 0.9402 for the amortized translator.

We propose that much of this gap comes from one mechanism. The loss is a weighted sum of several terms, typically an identity term, a face-parsing term and pixel or prior terms. During optimization the gradient of each term finds and exploits places where the proxy disagrees with the real engine. The fix we propose is to keep the real engine in the loop on a small call budget. Each loss component is scored by how often its proxy-predicted improvement holds up when the engine actually renders the current settings, and each component's weight is set by that reliability score. The engine renders from those checks also become fresh training pairs for the proxy near the optimization path. The claim is testable: at a fixed budget of real engine calls, this "verified proxy descent" should close most of the gap between per-target optimization and a translator trained on 200,000 images, without needing those images.

Core Claim

When real targets number fewer than 100 but the engine can be queried, the limiting factor is not how much data the proxy saw in total. The limit is that the proxy is wrong exactly where a multi-term loss pushes the parameters. Hand-set weights $\lambda_k$ on the loss terms cannot account for this, because one term's gradient may pass through the proxy faithfully while another's (say, a pixel term sensitive to high-frequency shading that the proxy blurs) points somewhere the engine does not follow.

We therefore replace the fixed weights with reliability-scaled weights. Let $\theta$ be the engine parameters, $F_1,\dots,F_M$ an ensemble of proxies, $L_k$ the $k$-th loss component, and $g_{k,m} = \nabla_\theta L_k(F_m(\theta))$ its gradient through member $m$. The effective weight is

$$w_k = \lambda_k \cdot \rho_k \cdot \frac{\lVert \bar g_k \rVert}{\frac{1}{M}\sum_m \lVert g_{k,m}\rVert},$$

where $\bar g_k$ is the ensemble-mean gradient. The final ratio is 1 when the members agree on direction and shrinks toward 0 when their gradients cancel. $\rho_k \in [0,1]$ is a running fraction of engine-verified steps on which the true change in $L_k$, computed from the real render $R(\theta)$, had the same sign as the proxy-predicted change. Ensemble agreement is cheap and available at every step. Engine verification is expensive, so it runs every few steps, and its renders are added to a small buffer used to fine-tune the proxies locally.

Relation to Prior Work

Our work builds on proxy-based inverse rendering. The Imitator/Translator line (Face-to-Parameter Translation for Game Character Auto-Creation; Fast and Robust Face-to-Parameter Translation for Game Character Auto-Creation; Neural Rendering for Game Character Auto-Creation) trains a frozen proxy of a commercial game engine. It then optimizes face sliders under identity, parsing and loop-consistency losses with fixed weights and a PCA subspace built from 18,000 CelebA faces. We reuse its proxy-plus-composite-loss structure and its benchmarks. We differ in three ways: the engine is never fully replaced, the loss weights are not fixed, and we target the regime where the large real corpus that the PCA basis and the translator require does not exist. Li et al. (Deep Polarization Cues for Single-Shot Shape and Subsurface Scattering Estimation) use a learned re-rendering network as a proxy for polarized subsurface scattering, trained on 117,000 synthetic scenes. Their setting is our second testbed, where we shrink the proxy's training set and ask whether engine verification substitutes for volume.

Other papers motivate the proposal by showing that the forward model matters inside a composite loss. Shi et al.'s projection loss for PET (A Novel Loss Function Incorporating Imaging Acquisition Physics for PET Attenuation Map Generation using Deep Learning) reduces projection-domain error by 16.3% by adding a physics term, but its projector is exact and analytical. Our question is what changes when that operator is a learned, fallible approximation. Similarly, the multi-term face objectives of Zhao et al. (Convincing 3D Face Reconstruction from a Single Color Image under Occluded Scenes) and Yu et al. (Face Super-Resolution Guided by Facial Component Heatmaps) rely on analytical renderers or no renderer at all, so they never meet the proxy-exploitation problem.

The small-sample literature constrains the design. Tartaglione et al. (Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data) show that high-capacity networks trained on a few hundred images learn acquisition shortcuts and that shallower models do better. That result argues against training any new network on the few real targets, and it is why our method trains nothing on them: the proxies are fit to engine renders only. 3FabRec (3FabRec: Fast Few-Shot Face Alignment by Reconstruction) reaches 8.29% landmark error from 10 labels, but only after pre-training on 2.1 million images. We treat that kind of external pre-training as unavailable in the target regime, except for fixed off-the-shelf loss networks such as the face recognizer inside the identity loss.

Research Program

The method has three parts.

The first is an ensemble of $M$ small proxies (we expect $M=5$) trained on random parameter sweeps through the engine. We keep each member modest in size, following the "deeper isn't better" evidence. Disagreement among members is the cheap signal of proxy error.

The second is per-target descent on the composite loss using the reliability-scaled weights above. Every $K$ steps the engine renders the current $\theta$. This updates $\rho_k$ and adds the render to a local buffer, and the proxies are briefly fine-tuned on that buffer. Steps whose engine check shows the total true loss went up are rejected and the step size is halved, which makes this a trust-region scheme driven by the engine.

The third is an analysis component. We will measure the exploitation gap, meaning the proxy-predicted loss minus the true-engine loss at convergence, separately for each loss component. This tests whether the F2P-versus-translator gap is really driven by exploitation and which components drive it.

Engine access is a practical risk. The commercial engine of Shi et al. may not be available to us. If it is not, we will build a parametric face renderer with deliberately non-differentiable stages (discrete hairstyle and accessory choices, hard visibility) and re-implement F2P and the translator on it. The published numbers in that case serve as context, not as a head-to-head comparison.

Baselines

The do-nothing reference is the mean parameter vector, or the PCA mean, rendered for every target. It shows how much of any verification score is just "generic face".

The direct predecessor is iterative descent through a single frozen proxy with the fixed composite loss of Shi et al. This is F2P (Face-to-Parameter Translation for Game Character Auto-Creation), reported at 0.6977 on LFW, 0.6042 on CPLFW and 0.5800 on CFP_FP in Neural Rendering for Game Character Auto-Creation. We run it with the same proxy architecture and synthetic sweep, the same loss terms, and the same per-target iteration count as our method. It receives no engine calls at test time.

The strongest existing method is the amortized Imitator/Translator with PCA whitening. It is reported at 0.9402 LFW, 0.7652 CPLFW and 0.8236 CFP_FP verification score in Neural Rendering for Game Character Auto-Creation, with more than 200,000 training images. We run it twice: once with full CelebA, as the data-rich upper reference, and once with only the same $N \in \{10, 30, 100\}$ real images our method sees. Its PCA basis is fit from those images as well, giving the honest few-target competitor. Both versions get the same tuning budget we give ourselves, a fixed grid over the loss weights and learning rate.

The black-box competitor is CMA-ES run directly on the real engine with the same composite loss. It is given exactly our total number of engine calls, counting the proxy-training sweep plus the test-time verifications.

Beating the field means three things. On $N \le 100$, our verification score must exceed both F2P and the few-image translator. It must exceed CMA-ES at an equal call budget. And it should recover at least half of the gap between F2P (0.6977) and the full translator (0.9402) on LFW.

Two ablations isolate the key ingredient: one using only the ensemble-agreement factor ($\rho_k \equiv 1$), and one using only engine verification ($M=1$). A third ablation applies the engine-driven step rejection while keeping the weights fixed. It tests whether the per-component reweighting matters beyond simply checking steps.

Why not just…?

Why not just optimize the real engine with a gradient-free search and skip the proxy? This is the most serious threat. If the engine is fast and the parameter space is small, CMA-ES on true renders may match us, because it never sees proxy error. Face slider spaces have hundreds of dimensions, so we expect gradient-free search to need far more calls. We have no reported result on Lacuna for CMA-ES on these benchmarks, however, so the equal-budget comparison above is decisive. If CMA-ES matches us at our budget, we will report that the proxy is unnecessary in this regime and restrict our claim to expensive engines such as the subsurface path tracer.

Why not just train a bigger, more accurate proxy on more synthetic sweeps? Synthetic queries are cheap for game engines. Optimization still searches for wherever the proxy is wrong, so global accuracy does not guarantee accuracy along the descent path. We test this by scaling the sweep from 10k to the 117k scale used by Li et al. and tracking the per-component exploitation gap. If the gap vanishes with scale, our method's value reduces to cases where engine calls are costly.

Why not just keep the parameters near the training distribution with a PCA prior and small steps? Shi et al.'s subspace projection does exactly this. It needs a large real corpus to build the basis, and it cannot tell which loss term is misbehaving. The fixed-weight-with-step-rejection ablation, with a PCA prior fit from the $N$ images, measures how much of our gain such constraints already capture.

Why not just train a translator on the few real images? The small-data evidence from Tartaglione et al. predicts that this overfits, and the few-image translator baseline tests the prediction directly.

Proposed Experiments

On faces, we draw $N \in \{10, 30, 100\}$ real target images and evaluate verification score on LFW, CPLFW and CFP_FP. We also use synthetic targets rendered from known parameters, which let us measure parameter recovery error directly, alongside the exploitation gap and the count of engine calls.

On translucent materials, we follow Li et al.'s setup. A path tracer serves as the engine, and the proxy training set is shrunk from 117k to 1k–10k scenes. We report surface normal MAE against their reported 0.0651, together with scattering-parameter error on held-out synthetic scenes.

The decisive checks are two. First, the per-component exploitation gap for F2P should be large and concentrated in specific terms, and it should shrink under our weighting. Second, our method should beat CMA-ES and the few-image translator at matched engine calls and matched $N$. If the exploitation gap is small for F2P, the core claim is wrong and the F2P-to-translator gap has another cause.