Matching at the Right Depth: Region-Adaptive Feature Levels for Detecting and Suppressing Janus Duplicates in Sparse-View 3D

Many current systems make 3D objects by first generating a handful of views with an image generator and then fitting a 3D model to those views. The generator has seen mostly front-facing photos, so it tends to paint a face or a beak onto the back of an object. This is the "Janus problem", named after the two-faced Roman god. Errors that small cause harm too. Features drift slightly from view to view, and the 3D fit then fills in floating blobs and blur. Existing fixes compare views using one fixed kind of feature everywhere. Some use raw pixels or perceptual distance. Others use one hand-chosen layer of a diffusion model, a generator that learns to turn random noise into images step by step. Our claim is that no single feature level can see both failures. Pixel-level and edge-level features give precise point matches on rigid, textured surfaces, but they cannot tell that a back-view face "is" the front-view face. High-level semantic features can make that judgment, but they are too blurry to pin geometry down.

We propose a cross-view consistency signal that does two things. First, for each image region it picks the feature level whose matches across views are most unambiguous, using a cheap peakedness test with no training. Second, it uses the semantic level for a separate job: flagging Janus duplicates, meaning parts that match strongly somewhere the 3D geometry says they should not exist. The same module serves as a drop-in loss in text-to-3D optimization and as a per-pixel trust weight when reconstructing from generated sparse views. The test is direct. It should reduce Janus duplicates that reconstruction-based consistency scores cannot see, without the oversmoothing of uniform regularizers.

Core Claim

Two pieces of prior evidence point the same way. In 2D matching, the best feature level depends on content. Learning to Compose Hypercolumns for Visual Correspondence (DHPF) trains small gating networks that switch ResNet layers on and off per image pair. It found that cars favor low-level edge layers and animals favor high-level semantic layers, reaching 90.7% PCK (percentage of correct keypoints) on PF-PASCAL. Diffusion Hyperfeatures learns one global mixing of Stable Diffusion features across layers and denoising timesteps. It reports that middle layers and early timesteps carry most semantic correspondence (72.56% [email protected] on SPair-71k versus 68.33% for DINOv2). Both stay in 2D, and both choose levels per image pair or globally rather than per region.

In 3D, Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences (CorrespondentDream) is the closest existing work. It penalizes disagreement between motion predicted by the 3D model's depth and correspondences taken from fixed upsampling layers of the diffusion network. Its authors note that those correspondences become unreliable on textureless and shiny surfaces. MIRROR rectifies diffusion latents toward depth-tracked neighbors uniformly over all pixels, and only on fixed-elevation circular orbits.

Our claim has two parts. Choosing the feature level per region by match peakedness yields a more reliable geometric consistency signal than any fixed level. A semantic-level duplicate test then catches Janus failures that a geometric or photometric round-trip check cannot catch. The second part matters because a consistently two-faced object is perfectly reconstructable. It can therefore score well under Carve3D's round-trip reconstruction metric while being plainly wrong.

Mechanism

Let $\ell$ index a candidate feature level: a (layer, timestep) pair of a frozen diffusion network, plus one pixel-level option. For a pixel $p$ in view $a$, the current 3D estimate gives a depth interval. Projecting that interval into view $b$ gives a short epipolar segment $S$, the line along which the matching point must lie. Let $s_\ell(q)$ be the cosine similarity at level $\ell$ between $p$ and a candidate point $q$ on $S$. We score each level by how decisive its best match is:

$$\rho_\ell(p) = \frac{\max_{q \in S} s_\ell(q)}{\max_{q \in S \setminus N(q^\ast)} s_\ell(q)}$$

Here $q^\ast$ is the best match and $N(q^\ast)$ is a small neighborhood around it, excluded so that the denominator is the best competing match. We pick $\ell^\ast(p) = \arg\max_\ell \rho_\ell(p)$, smoothed over superpixels. This is a modern version of the classic rule from Tomita et al. and Krol et al.: choose the attribute whose distribution has the deepest valley between peaks. If $\rho$ is low at every level, the region abstains and receives near-zero weight. This is intended to cover the shiny and textureless cases where CorrespondentDream's fixed layers inject noise.

The consistency loss is CorrespondentDream's depth-versus-correspondence disagreement, computed at $\ell^\ast$ and weighted by $\rho_{\ell^\ast}$.

The Janus test runs separately, at the most semantic level. It searches the whole target view rather than the epipolar segment. A pixel is flagged when it finds a strong, unique match in view $b$ at a location that the rendered depth marks as a different 3D surface point, and the original point is predicted to be hidden in view $b$. Bilateral symmetry produces the same pattern legitimately: a left eye matches a right eye. We therefore exempt pairs explained by a single mirror plane fitted to all matched 3D pairs. Flagged regions receive a penalty in optimization, or are down-weighted and re-generated in the reconstruction setting. Features are taken from one noised forward pass at a few fixed timesteps rather than DDIM inversion, the deterministic procedure that recovers the noise behind an image. Inversion costs 6.6 s per pair in Diffusion Hyperfeatures.

Research Program

We first build the selector and duplicate test as a frozen, training-free module on top of an existing diffusion network. We then use it in two settings.

Setting A: text-to-3D optimization. We replace CorrespondentDream's fixed-layer correspondence loss inside its own score-distillation loop, holding everything else fixed. Score distillation optimizes a 3D model so its renders look plausible to a 2D diffusion model.

Setting B: reconstruction from generated views. We take multi-view sets from the generators MIRROR uses (VideoMV, SyncDreamer, MVD-Fusion). We estimate depth as MIRROR does, with Depth-Anything-V2. We reconstruct with NeuS or 3D Gaussian Splatting, an explicit 3D representation made of many small colored blobs. The module's weights and Janus flags act as per-pixel trust. This plays the role of CAT3D's distance-based weighting in CAT3D, but driven by evidence of consistency rather than distance from the inputs.

Analysis. For each benchmark category we log which levels the selector chooses. The DHPF pattern predicts edge levels for rigid man-made objects and semantic levels for animals. If the selections come out uniform, the adaptive claim is weakened regardless of downstream numbers.

Baselines

Do nothing. The naive reference is the unmodified pipeline. In Setting A this is the multi-view diffusion prior with score distillation alone. In Setting B it is the raw generated views reconstructed with uniform weights.

Closest method. CorrespondentDream with its published fixed upsampling layers is the strongest competitor in Setting A, because it is the same loss with a fixed rather than adaptive feature level. We run it with identical prompts, seeds, optimization steps and diffusion prior, and we give it the same tuning budget for its loss weight. No reported result was found on Lacuna for CorrespondentDream on the DreamFusion prompt set or T3Bench. We will therefore rerun it rather than quote it.

Training-free rectification. MIRROR is the strongest Setting B competitor. Applied to VideoMV on T3Bench, it raises PSNR (peak signal-to-noise ratio) from 21.67 to 24.82 and SSIM (structural similarity) from 0.814 to 0.898, at 2.6–2.9 times the inference time. We run both methods on the same generated sets with the same depth estimator and report wall-clock time. Beating MIRROR means matching or exceeding 24.82 PSNR on T3Bench with VideoMV, while lowering the Janus rate and staying within MIRROR's time overhead.

Round-trip consistency. Carve3D is the reference for reconstruction-based consistency. Its reinforcement-learning fine-tuning lowers the round-trip score (MRC) from 0.0892 to 0.0606 on the 415-prompt DreamFusion set. We use its MRC metric as an evaluation axis, and we also use MRC itself as a per-pixel weight to form an ablation.

Object-level geometry. CRM's reported 0.0094 Chamfer distance and 16.22 PSNR on Google Scanned Objects (GSO) anchor geometry quality; see CRM.

Ablations. These isolate the key ingredient by holding the loss fixed and changing only the feature source:

  • the single best fixed level, chosen on validation data;
  • the fixed global learned mixture of Diffusion Hyperfeatures;
  • DINOv2 features;
  • an oracle that picks the best level per category in hindsight.

A last ablation removes the duplicate test while keeping the selector.

Why not just…?

Why not just use a good static mixture of features, like Diffusion Hyperfeatures' learned weights or DINOv2? This is the real threat. Hyperfeatures already gains 14% over the best raw layer with a single global mixture. Per-region selection may add little on top of that. The fixed-mixture and oracle ablations settle it. If the fixed mixture comes within noise of our selector on Janus rate and Chamfer distance, we drop the adaptivity claim and report the duplicate test as the contribution. If the oracle beats both, adaptivity matters but our peakedness rule is not capturing it.

Why not just penalize round-trip reconstruction error, as Carve3D does? A reconstructor can faithfully fit an object with two faces. The round-trip error is then low even though the shape is wrong. Carve3D's own reward is uniform LPIPS (a learned perceptual image distance) over all pixels, with no notion of which parts should be unique. We test this directly. On a human-labeled subset, we compare MRC between Janus and non-Janus outputs. Our hypothesis predicts that MRC barely separates the two, while the duplicate test does.

Why not just fix the prompt with Perp-Neg? Perp-Neg removes the "front view" direction from the diffusion guidance. It raised back-view success for a panda from 14.6% to 40.4% but still failed in most cases. Prompt-level fixes also cannot see geometric concavities, as CorrespondentDream documents. Perp-Neg is complementary rather than competing, so we report it alone and combined with our loss.

Why not just use generic geometric regularizers? RegNeRF removes floaters with depth smoothness and reaches 18.89 PSNR on 3-view DTU. Its authors note that this oversmooths fine geometry, and it has no notion of duplicated parts. We include it in Setting B to check that our gains are not simply smoothing.

Proposed Experiments

Setting A. We use the 415-prompt DreamFusion set and the 100-prompt T3Bench. Metrics are:

  • MRC;
  • a Janus rate from blinded human annotation on a fixed subset;
  • an independent duplicate count from an off-the-shelf face and part detector applied to orbit renders, not using our features, to avoid circularity;
  • runtime per asset.

Setting B. We use T3Bench for multi-view consistency (PSNR and SSIM, following MIRROR) and GSO's 100 household objects for geometry (Chamfer distance and volume IoU against ground truth, following MIRROR's protocol). To test the trajectory restriction MIRROR inherits, we add GSO renders with varying elevation.

Real sparse-view check. A secondary experiment asks whether the selector helps on real images, where Janus is absent. We apply the weighting to 2-view LLFF Gaussian Splatting, where HeroGS reports 18.78 PSNR.

Decisive checks. The project's central claims stand only if three things hold:

  1. Our selector beats the fixed-mixture ablation on Janus rate or Chamfer distance by more than seed variance.
  2. The duplicate test separates human-labeled Janus outputs from non-Janus outputs where MRC does not.
  3. False Janus flags on symmetric categories such as chairs and cars stay below a set rate.

Abstention on specular GSO objects should correlate with lower geometry error than fixed-layer CorrespondentDream on the same objects. If it does not, the reliability argument for adaptive selection fails.