Evolving Anisotropic Reflectance Code with Rendering Feedback

Most methods that recover 3D shape and material from photographs assume that a surface reflects light the same way no matter how it is rotated about its own normal. This is the isotropic reflectance assumption. It makes the inverse problem tractable, but it breaks on brushed metal, satin, CDs and woven cloth. On these materials, highlights stretch into streaks, and a pipeline that assumes isotropy explains the streaks by bending the surface. The current fix is to swap in one hand-chosen richer family of reflectance functions, usually anisotropic spherical Gaussians (smooth lobes on the sphere of directions with separate widths along two axes). The alternative is a black-box neural network that fits clean data well and noisy real data poorly.

We propose a third route. A language model writes and rewrites the reflectance function itself as a short program. Each candidate program is scored by fitting its parameters with gradient descent in a differentiable renderer. The language model is then shown where the fitted render still disagrees with the photographs, as per-pixel error maps split by light direction, and it proposes the next edit. Programs that violate basic optics are rejected before scoring. The claim is concrete. A code-refinement loop whose fitness is itself a continuous optimization, and whose feedback is spatial rendering error rather than a pass/fail test, can find compact, physically valid anisotropic reflectance kernels. Dropped into an existing photometric stereo pipeline, these kernels should reduce surface-normal error on anisotropic metals relative to both the fixed anisotropic spherical Gaussian family and the best known analytic model.

Where this sits

Two bodies of work meet here, and neither covers the combination.

On the reflectance side, DANI-Net replaces isotropic lobes with anisotropic spherical Gaussians inside an uncalibrated photometric stereo network and reports 6.54° mean angular error on DiLiGenT. Spec-Gaussian does the same for view-dependent color in Gaussian splatting. GlossGau builds an anisotropic microfacet normal distribution from warped Gaussians. All three motivate this proposal, because they show that isotropy is the bottleneck. They also constrain it: each fixes the functional form in advance and only tunes its parameters, so a reflectance profile outside that form cannot be expressed.

On Neural BRDFs supplies the tension we exploit. On clean measured data a single network beats the analytic Disney model (52.35 vs 46.46 PSNR on MERL), but on real multi-view captures the gap nearly closes (42.35 vs 41.89 on DiLiGenT-MV). Unconstrained networks also do not learn reciprocity without re-parameterization. Compact symbolic forms that are richer than hand-designed ones are the natural thing to look for in that gap.

On the code-refinement side, $A_2$DEPT supplies the search method. It evolves whole programs in a tree using simulated annealing and several mutation types, with a scalar fitness from solver runs. POLO guides edits with profiler output that tells the model where the cost lies. Our residual maps play the analogous role: they tell the model where the error lies. MultiMat is the closest graphics analogue. It builds procedural material graphs with visual feedback and backtracking, cutting KID from 14.155 to 2.365 against text-only generation. However, it edits graphs of 2D texture maps for a fixed PBR shader and never touches the shading math or fits real anisotropic measurements.

Our departure from all of these is a two-level loop. The outer loop edits code. The inner loop is a gradient fit, so each program's fitness is the best it can do after optimization. The feedback given to the model is structured by light and view direction, which is where anisotropy shows up.

Core Claim and Mechanism

A candidate is a single differentiable function with a frozen signature. It takes the normal, a tangent angle, the incoming and outgoing directions, and a small parameter vector, and returns RGB reflectance. Freezing the signature is deliberate. The "partial migration" failures documented in $A_2$DEPT, where a change to one component breaks another, cannot occur when the only thing being edited is one self-contained kernel.

For each candidate we fit per-pixel parameters, including the tangent angle, on a training subset of lights. The candidate's score is

$$ S(p) = L_{\text{held}}(p) + \beta \, |\theta_p|, $$

where $L_{\text{held}}$ is the photometric error on held-out lights after fitting, $|\theta_p|$ is the number of per-pixel parameters of program $p$, and $\beta$ trades accuracy against compactness. Scoring on held-out lights is what keeps the search from rewarding programs that memorize sensor noise, which is the failure seen in neural BRDFs.

Before scoring, a numerical gate rejects programs that fail two physical tests. The first checks reciprocity by comparing the output with incoming and outgoing directions swapped, over sampled pairs. The second is a Monte Carlo furnace test: the integrated reflectance, weighted by cosine, must not exceed one. This plays the role that Alive2 plays in Taneja et al.. It is weaker, because the check is statistical rather than a proof, and we say so.

The model receives three kinds of feedback: the score, the gate verdict, and residual maps binned by the angle between the half-vector and the tangent. The binning makes elongated-highlight errors visible as a pattern rather than a number.

Research Program

Stage one isolates reflectance. On objects with ground-truth normals, we freeze geometry and search only over kernels. This removes the geometry–reflectance ambiguity, in which a kernel could look good by letting the normals absorb the error.

Stage two replaces the anisotropic spherical Gaussian block in DANI-Net with the best discovered kernels and trains normally under uncalibrated lighting. This tests whether better reflectance code actually buys better shape.

Stage three transfers kernels found on one set of objects to held-out objects of the same material class without further search. This asks whether the loop discovers material models or per-object fits.

Throughout, we log the best score against iteration. Yan et al. found that compiler feedback plateaus after a few rounds, and we want to know whether rendering feedback does the same.

Baselines

The do-nothing reference is DANI-Net with its isotropic lobes, together with the Position Threshold heuristic from the DiLiGenT evaluation. That heuristic discards the brightest and darkest 40% of pixel intensities and was found to rival complex models on standard objects. Both run on identical images and splits.

The strongest existing method is DANI-Net with anisotropic spherical Gaussians. It reports 6.54° mean angular error on DiLiGenT. For the metallic objects of DiLiGenT10², the file gives no per-object numbers (no reported result found on Lacuna for DANI-Net per-object error on DiLiGenT10² metals), so we rerun it ourselves with its published three-stage schedule. Beating it means lower normal error on the aluminum, copper and steel objects, with no regression on DiLiGenT.

The most dangerous baseline is an analytic library. We fit each of several textbook anisotropic models in the same inner loop, with the same held-out scoring and the same compute, and pick the best per object by $S$. If the evolved kernels do not beat the best library model, the loop has only rediscovered known optics.

A purely neural reflectance network with the half-angle re-parameterization from On Neural BRDFs is included at matched parameter count.

For rendering, Spec-Gaussian's 37.70 dB against 33.82 dB for standard splatting on its anisotropic synthetic scenes is the reference point. Evolved kernels substituted for its appearance field should match that figure while being explicit reflectance functions.

The key ablation replaces the binned residual maps with the scalar score alone, under equal LLM calls. A second ablation removes the physics gate. A third runs classical genetic-programming symbolic regression over the same operators under an equal count of evaluated programs. Every search method receives the same number of inner-loop fits.

Why not just…?

Why not just keep anisotropic spherical Gaussians? They may be enough, and stage two measures exactly that. The reason to doubt it is that DANI-Net itself flags translucent and exotic materials as failures. A fixed lobe family cannot change shape when the residual maps show structured error. If evolved kernels only tie with anisotropic spherical Gaussians, we report that isotropy's replacement is already adequate for these benchmarks.

Why not just pick the best known analytic anisotropic model? This is the real threat. Much of the gain may come from simply allowing anisotropy, and textbook models already do that. The library baseline, fit under the identical inner loop, settles it. We will also inspect whether the evolved winners are textbook models in disguise.

Why not just use a neural reflectance network? The evidence from On Neural BRDFs is that its advantage over analytic forms nearly vanishes on real captures, and that it needs architectural surgery to respect reciprocity. Our held-out-light comparison at matched size tests whether compact code generalizes better on real metals.

Why not just run symbolic regression without a language model? Genetic programming can search expression spaces without an LLM. Two things it lacks are the ability to read a residual map and a prior over sensible optics. The equal-budget genetic-programming baseline and the scalar-feedback ablation separate these two contributions. If genetic programming matches us, the language model is unnecessary and we will say so.

Decisive Checks

The proposal succeeds if three things hold together. First, in stage one, evolved kernels achieve lower held-out photometric error than the best library model on DiLiGenT10² metals at comparable parameter count, while passing the physics gate. Second, in stage two, they lower normal error relative to anisotropic-spherical-Gaussian DANI-Net on those metals without worsening DiLiGenT beyond its 6.54°. Third, the residual-map ablation shows a gap, which would indicate that spatial rendering feedback is the working ingredient.

It fails informatively in two ways. If the library baseline wins, known optics suffice. If the plateau arrives within a few iterations as in Yan et al., rendering feedback adds little beyond the first edit. Token cost is reported per object against the inner-loop compute, since Adversarial Review shows that multi-agent loops can cost several times more than single-shot generation.