Is the Janus Head Sustained by Curl? Testing Score Distillation as a Gradient Flow

Text-to-3D systems built on 2D image generators often produce the "Janus" artifact: a dog with a face on the front and another on the back of its head. The usual explanation is that a frozen 2D diffusion model, which turns random noise into images by removing noise step by step, judges each rendered view on its own and wants a face in every view. Fixes therefore reweight camera poses, edit view prompts, clip large gradients, or warp gradients between views. This note asks a more basic question about the optimization itself. The standard update, score distillation sampling (SDS), takes the 2D model's denoising residual on a rendered view and backpropagates it into the 3D scene. That update is the gradient of a single scalar objective only if the 2D model's output is a true score, meaning the gradient of a log-density. Earlier work shows that standard diffusion networks do not output true scores. They learn fields with a rotational, curl-like component that is the gradient of nothing.

If the 2D field rotates, the 3D optimization is not descending any objective. It can then settle at equilibria that no objective would pick. The claim tested here is that some Janus heads are fixed points of this kind, held in place by the rotational part of the distilled update, and that the rest are true local minima of a conservative objective, as current certainty-based and pose-based accounts assume. These two causes call for different fixes. The proposal builds a cheap measure of rotation for the 3D update, tests whether it predicts Janus onset beyond gradient magnitude and cross-view disagreement, and then intervenes on it causally.

Core Claim

Write the SDS update on 3D parameters $\theta$ as $g(\theta)=\mathbb{E}_{c,t,\epsilon}[w_t R^\top(\hat\epsilon(x_t)-\epsilon)]$. Here $x=\text{render}(\theta,c)$ is the image from camera $c$, $R=\partial x/\partial\theta$ is the renderer's Jacobian, $x_t$ is the image noised to level $t$ with scale $\alpha_t$, $\hat\epsilon$ is the 2D noise predictor, and $w_t$ is a weight. Differentiate $g$ and keep only the antisymmetric part. The renderer's second-derivative terms are symmetric and drop out, which leaves

$$\tfrac12\big(\nabla_\theta g-\nabla_\theta g^\top\big)=\mathbb{E}\Big[w_t\alpha_t\,R^\top\,\tfrac12\big(D\hat\epsilon-D\hat\epsilon^\top\big)\,R\Big],$$

where $D\hat\epsilon$ is the Jacobian of the noise predictor with respect to its input image. The rotational part of the 3D update is therefore exactly the 2D network's Jacobian asymmetry, seen through the renderer and averaged over cameras.

Two consequences follow. First, disagreement between views does not by itself create rotation. A sum over cameras of gradients of different per-view energies is still a gradient. So the account of multi-modal head densities in LCGen describes a conservative landscape with bad minima. Second, rotation enters only through the non-conservativeness that On Investigating the Conservative Property of Score-Based Generative Models documents in 2D networks. That paper measured the asymmetry on CIFAR-scale images and regularized it with a Hutchinson estimator, which approximates a matrix trace or product with random probe vectors. Neither that paper nor any 3D paper in the file asks what the asymmetry does once it is pulled back into a scene.

The proposal turns the decomposition into a measurable ratio. Let $\kappa=\|\text{asym}(\nabla_\theta g)\|/\|\text{sym}(\nabla_\theta g)\|$, estimated with random probe pairs $(v,w)$ from $v^\top J w - w^\top J v$. Each probe needs one forward-mode and one reverse-mode pass through the renderer and the U-Net. It then asks whether $\kappa$ is large near Janus states and whether removing it removes them.

Positioning

The closest work splits into three groups. The first constrains the method, the second motivates it, and the third has to be beaten.

Theory that constrains the method. Score-based Generative Modeling Secretly Minimizes the Wasserstein Distance bounds the 2-Wasserstein distance by the square root of the denoising loss, under a one-sided Lipschitz condition on the score network. It assumes the network is a well-behaved score and studies flat data, not distillation through a renderer. The analysis here supplies the missing piece for distillation: when the field rotates, the update is not a gradient flow at all, so bounds of the Kwon type and of the gradient-flow type cannot apply. Improving Flow Matching by Aligning Flow Divergence shows the analogous gap in flow matching. Matching velocities without matching their divergence lets the probability path drift in total variation, the largest difference in probability the two distributions assign to any event. Its divergence term is a candidate second regularizer here, but the main object is the antisymmetric part, not the trace.

Mismatch accounts that motivate the question. RecDreamer treats Janus as a biased pose marginal and reweights scores toward uniform pose. LCGen treats it as spurious peaks produced by discrete view conditioning. L²-VSD traces collapse to a one-step lag in variational score distillation, in which a small LoRA tracking network follows the scene's rendered distribution but lags it, and keeps only the first-order lookahead term. All three implicitly treat distillation as descent on some objective. This proposal tests that assumption directly.

Heuristic fixes that are competitors. D-SDS observes that Janus-inducing gradients have unusually large magnitude, clips them, and edits view prompts. GSD anchors noise in 3D, warps gradients between views using rendered depth, and penalizes their variance. Gradient magnitude and cross-view variance are both plausible confounds of $\kappa$. Each must be controlled for explicitly.

Research Program

Stage 1: build and validate the estimator. The estimator computes $\kappa$ for NeRF and 3D Gaussian Splatting scenes using Stable Diffusion as the prior. It is checked on a constructed case where the answer is known. Replace $\hat\epsilon$ with the gradient of an explicit scalar energy network, which is conservative by construction, and confirm that $\kappa$ is near zero. Then confirm that $\kappa$ matches the 2D asymmetry measured directly on renders.

Stage 2: diagnose. Optimize animal and character prompts, including the 70-prompt set used by D-SDS, across many seeds. Log $\kappa$, gradient norm, and GSD-style warped-gradient variance along each trajectory. Label Janus onset with a DINOv2 front/back classifier of the kind RecDreamer uses, and spot-check the labels by hand. The diagnostic question is whether $\kappa$ rises before onset and predicts it after conditioning on the two confounds.

Stage 3: test equilibria. Take converged Janus states and ask whether they are local minima of a conservative surrogate. Rerun the optimization from those states with the conservative component alone, either from a conservative-by-construction energy prior or with the full gradient through the U-Net Jacobian, and see whether it escapes. If it escapes, rotation was holding the state in place. If it stays, Janus there is a genuine minimum, which supports the certainty and pose-bias accounts.

Stage 4: intervene. Two interventions target the rotation. On the prior side, fine-tune a LoRA on the diffusion model with the asymmetry regularizer from the conservative-property paper and then distill. On the distillation side, add a Hutchinson curl penalty, evaluated every few steps to bound its cost. The cost matters because RecDreamer already takes about 297 minutes per asset against VSD's 180 on an RTX 4090.

Baselines

Naive reference. The naive reference point is DreamFusion-style SDS with view-dependent prompts (DreamFusion). It runs with the same renderer, prompt set, seeds and step count as every other method. It anchors the Janus rate and the natural level of $\kappa$.

Strongest existing Janus method. The strongest competitor on Janus is RecDreamer, because it directly rectifies the pose bias that a skeptic would say fully explains the artifact. It runs from its released configuration on the same prompts, and its wall-clock time is reported alongside ours. For RecDreamer's quantitative Janus rate there is no reported result found on Lacuna. The file gives only its runtime: about 297 minutes, against 180 for VSD. Beating it means a lower Janus rate on the non-canonical and multi-object prompts where pose-marginal reasoning should break, without a worse T3Bench quality score. Matching it at lower cost would also count.

Quality reference. L²-VSD reports an average CLIP similarity of 0.285 on its 20-prompt benchmark. We rerun it on those prompts as a guard against interventions that fix geometry by blurring texture. D-SDS and GSD are included at their published settings because they control the two confounds.

Ablations isolating rotation. The key ablation is a magnitude-matched control. It clips gradients so that their norm schedule matches the curl-penalized run, which separates rotation from size. A second control fine-tunes the LoRA without the asymmetry term, which separates conservativeness from fine-tuning in general. A third keeps the full U-Net-Jacobian gradient, which is conservative by construction. Every method gets the same tuning budget of a fixed number of hyperparameter trials, chosen on a held-out set of 10 prompts.

Why not just…?

Why not just use the full gradient through the U-Net, which is already conservative? This is the most serious alternative. If the full gradient cured Janus cheaply, the curl penalty would be pointless. DreamFusion dropped the U-Net Jacobian because it made optimization worse, which suggests that conservativeness bought this way costs signal. Stage 3 and the third ablation measure that trade-off directly. If the full gradient wins, the conclusion is still the central one, that rotation matters, and the method simplifies to the full gradient.

Why not just clip large gradients, as D-SDS does? Rotation and gradient magnitude may be the same thing in disguise. The magnitude-matched control decides this. If clipping to the same norm schedule removes Janus equally well, the curl story adds nothing and we will say so.

Why not just penalize cross-view gradient disagreement, as GSD does? The decomposition shows that disagreement between views creates no rotation. Variance and $\kappa$ therefore measure different things. GSD also depends on usable early depth maps. The Stage 2 regression asks which signal predicts onset, and the comparison on degraded-initialization prompts asks which one still works when early depth is poor.

Why not just use a multi-view diffusion prior such as MVDream, a leader on T3Bench? A multi-view prior avoids the question by training on 3D data. The question here concerns distilling 2D-only priors, and $\kappa$ can be computed for any prior. Testing a multi-view prior is an extension, not a substitute.

Proposed Experiments and Decisive Checks

Generation runs on T3Bench. Its 161-view quality scores are smoothed across views to penalize inconsistency, and its scores correlate with human ratings at Spearman correlation above 0.75. Generation also runs on the DreamFusion 415-prompt set and on stress prompts taken from LCGen's failures, such as "an upside-down lion" and "three corgis". The metrics are Janus rate, pose entropy, CLIP R-precision, and multi-view reconstruction consistency (MRC), all with seeds reported.

MRC is borrowed from Carve3D. It reconstructs a scene from the generated views with a large reconstruction model, re-renders it from those views, and scores the LPIPS perceptual distance between the two image sets. Carve3D reports 0.0606 against 0.0892 for its base model on the DreamFusion set. That comparison is for a fine-tuned multi-view generator, so it serves as scale context, not as a baseline to beat.

The hypothesis is supported if three things hold. First, $\kappa$ predicts Janus onset after controlling for gradient norm and cross-view variance. Second, Janus states escape under conservative flow in Stage 3. Third, curl reduction lowers Janus rate on non-canonical prompts more than the magnitude-matched control does.

The hypothesis is refuted if $\kappa$ adds no predictive power, or if Janus states stay put under conservative flow. That outcome would also be useful. It would establish that Janus is a conservative-landscape problem, which supports RecDreamer and LCGen and directs future effort to the objective rather than to the vector field.