Curvature-Calibrated Timescales for Variational Score Distillation

Text-to-3D methods such as variational score distillation train two models against each other at the same time. The first is a 3D scene. The second is a small critic: a low-rank adapter, a few trainable weights bolted onto a frozen image diffusion model so that it learns what the scene's current renders look like. The critic must keep up with the scene. When it lags, the scene chases a stale target and drifts toward blur and slow convergence. When practitioners speed the critic up by raising its learning rate, runs suffer "color explosions" and geometric collapse instead, as Advancing Text-to-3D Generation with Linearized Lookahead Variational Score Distillation reports.

We propose that this dilemma is a mis-set timescale separation, not a fundamental limit. Theory for leader–follower games says the fast player, here the critic, must contract toward its best response faster than the slow player moves. A single scalar learning rate cannot do that when the critic's loss has very different curvature across its coordinates. A step large enough to make flat directions keep up overshoots the stiff directions, and those stiff directions are where the explosions start. Our fix is to give every critic coordinate its own step size, inversely proportional to a cheap running estimate of that coordinate's curvature. Each coordinate then tracks its best response at the same rate, and the separation from the scene can be set by one interpretable constant rather than a ratio that must grow with the square of the condition number.

Core Claim

The critic update in variational score distillation is the fast half of a two-timescale leader–follower game. Its stability is governed by per-coordinate curvature, not by the overall learning-rate ratio. Write $y$ for the critic's adapter weights, $x$ for the scene parameters, $g(x,y)$ for the critic's denoising loss on current renders, and $y^*(x)$ for the critic that minimizes it. Near $y^*(x_t)$, the tracking error $e_t = y_t - y^*(x_t)$ of a critic updated with per-coordinate step sizes $D_\eta$ evolves approximately as

$$e_{t+1} \approx (I - D_\eta H_{yy})\, e_t - J^*\, \Delta x_t,$$

where $H_{yy}$ is the critic-loss Hessian, $J^* = \partial y^*/\partial x$ measures how much the ideal critic moves when the scene moves, and $\Delta x_t$ is the scene step.

A scalar step $D_\eta = \eta I$ must satisfy $\eta < 2/\lambda_{\max}(H_{yy})$ to avoid blowing up stiff directions. It then contracts the flattest directions only at rate $1 - \eta\lambda_{\min}$. This is the origin of the $\kappa^2$ ratio requirement in On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems, which proves single-loop two-timescale descent–ascent converges when the ratio scales with the squared condition number $\kappa$. Choosing $D_\eta = c\,\mathrm{diag}(\hat H_{yy})^{-1}$ instead gives every coordinate contraction near $1-c$, insofar as $H_{yy}$ is diagonally dominant. The scene's step then only needs to be small relative to $c$ and to the coupling $\|J^*\|$. The required separation no longer depends on how ill-conditioned the critic is.

The prediction is concrete. A curvature-calibrated critic can be made effectively much faster than the scalar-rate critic without color explosion. Doing so should reduce the critic-lag artifacts that motivate lookahead methods, at roughly the cost of standard variational score distillation.

Why This Is Different From What Exists

The setting comes from ProlificDreamer. That paper introduces variational score distillation with an on-the-fly adapter critic and reports 3D-FID 107.02 against 118.92 for plain score distillation sampling. It co-optimizes critic and scene with hand-set scalar rates.

The closest work is Lei et al.'s linearized lookahead method. It diagnoses the same critic lag and fixes it by reordering the updates so the critic steps first, then linearizing that lookahead with a Jacobian–vector product. It explicitly discards higher-order terms as destructive. We keep the original update order and change how fast each critic coordinate moves. Our use of curvature is also different in kind: it calibrates step sizes and does not add a Taylor correction to the gradient. That distinction is how we resolve the apparent conflict the file notes between Lei et al., who say second-order terms hurt, and 3DGS²-TR, which finds diagonal-Hessian steps with a Hellinger trust region beat Adam on Gaussian-splat reconstruction. Both can be true if curvature helps as a step-size scale and hurts as an additive gradient term.

Adversarial Score Distillation recasts the critic as a Wasserstein discriminator with Lipschitz penalties. It uses standard first-order optimizers and never analyzes the rate ratio, so it motivates the game view but does not supply our mechanism.

On the theory side, three papers supply the frame:

All three treat step sizes as scalars and test on toy games or small GANs. We make the ratio per-coordinate and test it in a large-scale generative setting.

Dissecting adaptive methods in GANs shows Adam's benefit in GANs comes mostly from balancing update magnitudes between the two players. This constrains our design: our method must beat magnitude normalization, not just plain SGD.

Research Program

Toy game. We first establish the mechanism on a linear–quadratic leader–follower game whose follower Hessian has a controlled eigenvalue spread $\kappa_y$, the style of synthetic game used by Chae et al. We will derive the stability region of scalar versus diagonally calibrated follower steps. We will also measure the largest stable effective follower speed and the steady-state tracking error as $\kappa_y$ grows, including near-singular follower curvature, where we expect calibration to need a damping floor.

Curvature estimator. The critic loss is a mean-squared denoising error, so we estimate the diagonal of its Gauss–Newton matrix rather than the raw Hessian. Hutchinson's probe applied to the Gauss–Newton product, a random-sign vector $z$ elementwise times $J^\top J z$ computed with one forward-mode and one reverse-mode pass, gives an unbiased diagonal estimate that is nonnegative in expectation. This sidesteps the negative-curvature noise that the file flags for plain Hutchinson estimates. Because the adapter is small, the probe is cheap relative to the diffusion-model forward pass that score distillation already pays for. We refresh it every $K$ steps, smooth it with an exponential moving average, and clip it below at a damping value $\lambda$.

The method. The critic step becomes $c\,(\hat d + \lambda)^{-1} \odot \nabla_y g$, where $\hat d$ is the smoothed curvature estimate. The scene keeps its usual optimizer, with its learning rate tied to $c$ by a single ratio. In a secondary variant, the scene also receives diagonal-Hessian steps under a 3DGS²-TR-style Hellinger trust region when the scene is represented as 3D Gaussian splats.

Diagnostic. The central measurement is tracking error. Every few thousand steps we snapshot the scene, train a reference critic to convergence on that frozen snapshot, and measure the distance between the live critic's noise predictions and the reference critic's on held-out renders. This tests the causal story directly. If our method improves image quality without reducing tracking error, the story is wrong even if the numbers look good.

Baselines

Standard variational score distillation. This is the naive reference: ProlificDreamer's recipe with its published learning rates and Adam for both players. It defines what "lag" looks like.

Scalar timescale sweep. We raise the critic's Adam learning rate relative to the scene over a grid. This is the intervention Lei et al. report causes color explosion. It establishes the frontier our method must move past: at each effective critic speed, does a run survive, and what does it score?

Lookahead with linearization. This is the strongest existing method. On Lei et al.'s 20-prompt benchmark, the file reports average FID 284.06 against 301.54 for standard variational score distillation, and 10,000 against 15,000 steps to converge, at 2.7 versus 3.0 hours on an RTX 3090. The same table reports average CLIP similarity of 0.285 for the lookahead method against 0.324 for the baseline. We will report this oddity as given and recompute both numbers ourselves rather than rely on it.

Fair comparison means the same prompts, renderer, diffusion backbone, guidance scale of 7.5, three seeds per prompt, an identical hyperparameter-tuning budget of equal trial counts per method, and matched wall-clock time. Beating it means lower FID at equal wall-clock and no worse CLIP score, with a lower color-explosion rate.

Ablation isolating curvature. The key ablation replaces our curvature estimate with Adam's squared-gradient second moment, keeping every other part of the calibration identical. A second control randomly permutes our curvature estimates across coordinates, which preserves their distribution but breaks the correspondence. If the squared-gradient version matches ours, the claim that true curvature matters is false.

Why not just…?

Why not just use Adam with a bigger critic learning rate? Adam already rescales each coordinate. However, it rescales by recent gradient magnitude, and in a moving-target game that magnitude is dominated by lag, not stiffness. Large errors in flat directions get shrunk, while small-gradient stiff directions get large steps, which is the wrong way round. Lei et al. report exactly this failure, color explosion when the critic rate is raised. Jelassi et al. show Adam's GAN benefit is magnitude balancing, which does nothing about curvature. This is the alternative most likely to threaten us, and the squared-gradient ablation settles it directly.

Why not just run several critic steps per scene step? Multiple inner steps buy tracking with compute. Lin et al. found that single-loop two-timescale descent–ascent beat the multiple-ascent variant in speed and accuracy on robust classification. Multiple steps also do not fix the stiffness problem, since each inner step still faces the scalar stability limit. We include a $k$-step critic at matched wall-clock time as an additional run in the frontier comparison.

Why not drop the critic and use noise-free score distillation? Noise-free Score Distillation removes the destructive noise term and improves MS-COCO FID from 39.04 to 35.18 without any game. By its authors' account, though, it remains mode-seeking and multi-faced, and ProlificDreamer argues that any point-based loss shares this. Our proposal targets the variational regime where diversity requires a critic. We run noise-free score distillation as a quality reference and measure prompt-level diversity, so readers can see whether the game is worth its cost.

Why not use Lei et al.'s lookahead? Lookahead and calibration attack the same lag from different sides, and lookahead may already capture most of the gain. We therefore also test them combined. If calibration adds nothing on top of lookahead, we will report that as the outcome.

Proposed Experiments

The primary benchmark is Lei et al.'s 20-prompt comparison, scored with CLIP similarity, FID, steps to convergence, and wall-clock time. We add DreamFusion's prompt set scored with 3D-FID, for comparison against ProlificDreamer's 107.02. We add T3Bench for quality and alignment across single-object, surroundings, and multi-object tiers. No per-model T3Bench result was found on Lacuna, so we run all baselines there ourselves. Artifact metrics are mean rendered saturation, the fraction of runs that hit saturation blow-up or produce non-finite values, and tracking error over training.

Three checks are decisive:

  1. Frontier. Plot color-explosion rate against effective critic speed for scalar Adam, the squared-gradient ablation, and our method. The claim requires a clear rightward shift for ours alone.
  2. Mechanism. Tracking error must fall under calibration, and its reduction must predict the FID gain across prompts.
  3. Wall-clock. The probe overhead must not erase the gain at matched time.

If the toy game shows no stability advantage once $\kappa_y$ is realistic, or the squared-gradient ablation matches true curvature, we abandon the curvature claim and report the result as a negative finding about where critic lag comes from.