Early-Window Backtracking for the Janus Problem

Image generators that are asked to produce several views of one object often draw a face on the back of the head. This failure is known as the Janus problem. It happens because the underlying 2D model judges each view on its own and prefers the frontal features it saw most during training. Language models fail in a similar way when they reason. An early locally plausible mistake enters the context, and every later step is conditioned on it.

Language-model safety research has recently found a cheap fix for its version of this trap. Watch the generation as it unfolds. If it starts heading somewhere bad, intervene early, because the first few steps decide most of the outcome. We propose to test whether multi-view image generation has the same kind of early decision window. If it does, a cheap consistency check partway through generation could catch a Janus sample and restart it, or repair just the faulty view, long before the compute has been spent.

The concrete system has three parts. It starts with a multi-view diffusion model, meaning a generator that denoises random noise into several camera views of one object at once. Partway through denoising, it takes the model's current guess of the finished views and asks a fast 3D reconstruction network whether those views agree on one object. When they do not, it rolls back only the inconsistent view and resamples it. If the early guess predicts the final consistency well, this kind of early backtracking should match or beat "generate many, keep the best" at a fraction of the compute. If the early guess predicts poorly, the idea is wrong, and the experiments are designed to show that clearly.

Core Claim

Our claim is that Janus-type inconsistency is largely decided in the first fraction of the denoising steps. A geometric check on intermediate predictions can therefore play the role that step-level safety monitors play for reasoning models. The method borrows its structure from three safety papers. SafeThink monitors reasoning steps with a reward model and finds that steering only the first 1 to 3 steps recovers safety: attack success on HADES falls from 69.07% to 5.65%, while MathVista accuracy moves only from 65.20% to 65.00%. Backtracking Improves Generation Safety shows that erasing a bad prefix and restarting escapes the trap of conditioning on one's own early output. Any-Depth Alignment shows that a check reusing the model's own internal state can be far cheaper than running an external judge.

Two papers motivate the early-window hypothesis in vision. In-N-Out observes that views denoised from uncorrelated noise diverge early in their trajectories. The viewpoint-overfitting account in DreamBooth3D suggests that the frontal-feature injection is a coarse, layout-level decision, and diffusion models fix coarse layout early.

The reasoning-limitations literature constrains the design. Multimodal Language Models Cannot Spot Spatial Inconsistencies finds that GPT-5 with high reasoning reaches only 34.2% on detecting cross-view spatial impossibilities, against 84.8% for humans, and that GPT-4o as a judge is near chance. This rules out the natural design of asking a multimodal reasoning model "do these views agree?" The monitor must instead be geometric, built on reconstruction and re-rendering. Two observations make the link to reasoning more than an analogy. First, joint denoising across views is a sequential process in which early commitments compound, as in MME-Unify, which reports a cliff-like accuracy collapse by step 5 of visual chain-of-thought. Second, verbal reasoning is demonstrably the wrong verifier for this kind of error.

Research Program

The monitor builds on the reconstruction-consistency score of Carve3D. Carve3D reconstructs a 3D object from the generated views with a fast Large Reconstruction Model (LRM), re-renders it from the same cameras, and measures perceptual error. It uses that score as an offline reinforcement-learning reward. We instead compute the score during sampling, on the denoiser's one-step estimate of the clean views. At denoising step $t$, with noisy views $x_t$, predicted noise $\epsilon_\theta(x_t,t)$, and noise-schedule coefficient $\bar\alpha_t$, the estimate is the standard

$$\hat{x}_0(t) = \frac{x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}},$$

and the intermediate consistency score of view $v$ is

$$m_v(t) = \mathrm{LPIPS}\big(\hat{x}_0^{(v)}(t),\ R_v(L(\hat{x}_0(t)))\big).$$

Here $L$ is the LRM, $R_v$ renders the reconstruction from camera $v$, and LPIPS is a learned perceptual distance. Averaging $m_v$ over views gives the per-step analogue of Carve3D's sample-level score. Because Janus artifacts are often one view that disagrees with the rest, the per-view residual tells us where to intervene, not just whether to.

The first phase is pure measurement, and it is the hinge of the project. We sample many trajectories per prompt from a multi-view diffusion model of the Instant3D family used in Carve3D. We record $m_v(t)$ at a grid of steps and ask how well each early value ranks samples by their final consistency score and by human-labeled Janus status. The result is a predictiveness curve over denoising time. If rank correlation is already high at 20 to 30% of the schedule, an early window exists. If it only becomes high near the end, the analogy to SafeThink fails, and we report that as a negative result.

The second phase builds two interventions that use the window, in the spirit of SafeThink's satisficing threshold. In whole-sample restart, any trajectory whose early score exceeds a threshold is discarded and redrawn from fresh noise. This is early-abort best-of-N. In per-view backtracking, only the views with the largest residuals are re-noised back to an earlier step and resampled while the consistent views are held fixed. During this resampling, a view-specific negative prompt in the style of Perp-Neg is applied only inside the window. Perp-Neg removes only the part of a "front view" negative score that overlaps the "back view" positive score, rather than running it always-on.

The third phase characterizes cost. Every method is charged in denoising-step equivalents plus LRM calls, and the monitor is run at only one or two checkpoints so its cost stays small. We also check whether the gains stack on top of a Carve3D-fine-tuned model, which would make the method complementary to training rather than a substitute for it.

Baselines

The naive reference point is plain sampling from the base multi-view model with no monitor. On the 415 DreamFusion test prompts, Carve3D reports a mean reconstruction-consistency score of 0.0892 for its Instant3D base, where lower is better. We will re-measure this with our own LRM and renderer so that all numbers share one pipeline.

The strongest inference-time competitor is full best-of-N. It generates N complete samples and keeps the one with the lowest final consistency score. It uses the same verifier we use but has perfect information, so the only thing our method can win on is compute. The Lacuna file has no reported result for this baseline. We will run it on the same prompts and seeds at several values of N, then compare at matched total compute. Beating it means either lower final score and lower human-judged Janus rate at equal compute, or equal quality at no more than half the compute.

The strongest training-based reference is Carve3D's reinforcement-learning-tuned model, which reports a score of 0.0606 on the same prompt set and a 68.33% human preference over its base. It is not directly comparable, because it pays a training cost and risks the oversmoothing that the authors flag. We treat it two ways: as a ceiling to approach without training, and as a base model on which our backtracking should still add improvement.

The steering baseline is always-on Perp-Neg. Its paper reports that faithful back views rise from 14.6% to 40.4% on its view-faithfulness suite. We run it with a per-prompt tuning budget equal to the one we give our own thresholds.

Three ablations isolate the key ingredients. The first applies the same intervention at random or late timesteps rather than early ones; it tests whether timing matters. The second compares whole-sample restart with per-view repair; it tests whether localized backtracking matters. The third replaces the geometric monitor with a multimodal reasoning judge (GPT-4o or an open reasoning VLM) asked whether the views agree; it tests whether the geometric verifier is necessary.

Why not just…?

Why not just do best-of-N with the final consistency score? This is the real threat. If the early score does not predict the final score, best-of-N is strictly better and our method adds nothing. The phase-one predictiveness curve settles this before any intervention is built. The matched-compute comparison then settles whether per-view repair adds anything beyond early abort. We will report the crossover point honestly, including the case where the early window is too late to save meaningful compute.

Why not just fine-tune the model for consistency, as Carve3D does? Fine-tuning works, but it requires a reinforcement-learning loop with a reconstruction model in the reward. Carve3D notes that without regularization the model can collapse toward oversmoothed geometry. Supervised fine-tuning on 3D data is weaker still: Carve3D reports that its tuned model beats a model trained on ten times more supervised multi-view data. An inference-time method needs no retraining and can sit on top of a tuned model. The stacking experiment shows whether the two approaches are complementary or redundant.

Why not just use negative prompts or always-on Perp-Neg? Plain negative prompting strips core semantics when the negative concept overlaps the subject. Perp-Neg was designed to fix exactly that, but it still needs sensitive per-prompt thresholds and steers every sample, including ones that were already consistent. This mirrors Conditioned Activation Transport, which finds that ungated linear steering halves image fidelity, with CLIP score falling from 0.33 to 0.16, while gated steering preserves it at 0.32. Our gate is the geometric monitor itself. The comparison with always-on Perp-Neg measures whether gating preserves quality on prompts that never needed help.

Why not just ask a multimodal reasoning model to check the views? The evidence says this fails. Multimodal Language Models Cannot Spot Spatial Inconsistencies reports 34.2% for GPT-5 with high reasoning and near-chance for GPT-4o on detecting cross-view impossibilities. We include such a judge as an ablated monitor to confirm the failure in our setting. As a secondary sanity check, we run the reconstruction residual on the object-centric subset of that paper's Hypersim-derived inconsistency pairs to see whether it beats the reported reasoning-model accuracy. We expect this to be only partly informative, because the LRM is built for single objects, not indoor scenes.

Proposed Experiments

The main evaluation uses the 415 DreamFusion prompts. Metrics are final reconstruction consistency, a blind human count of extra faces and duplicated frontal features on a fixed random subset (human labels are needed because automated vision-language judges are unreliable here), and compute cost. The Perp-Neg view-faithfulness suite serves as a secondary check on whether the model honors non-canonical view requests. To check that consistency is not bought by blandness, we also report CLIP text-image score and a per-prompt diversity measure across seeds, since an oversmoothed object can trivially reconstruct well.

Three checks are decisive. The first is the shape of the predictiveness curve: rank correlation between early and final consistency should be high within the first third of denoising, or the hypothesis is refuted. The second is that early intervention beats late intervention at equal cost, which is the timing ablation. The third is that per-view backtracking beats both whole-sample restart and full best-of-N at matched compute without lowering diversity or CLIP score.

A positive result would show that the main lesson of early-window safety steering — commitments are made early, so check and correct early — holds for geometric consistency in generation. It would also show that the verifier must be grounded in reconstruction rather than in a reasoning model's verbal judgment.