Gray-Coded Virtual Projection for Multi-View Generation

Text-to-3D and image-to-3D generators often produce the "Janus" failure: an object with a face on the front, and another face on the back or sides. The cause is that a 2D image generator, asked what an object looks like from behind, has no firm notion of which side of the object it is drawing. Structured-light scanners faced a very similar problem decades ago. A projected sinusoidal stripe pattern tells a camera precisely where it sits within a stripe, but not which stripe. Metrology rigs solve this by also projecting a coarse Gray code: a stack of binary patterns in which the first pattern says which half of the scene a pixel belongs to, the next says which quarter, and so on. Adjacent codes differ in only one bit. Coarse bits fix the global index, and the fine sinusoid refines it. We propose to give a multi-view generator the same thing, as an extra output rather than a projector. Alongside RGB, the model generates, for every pixel in every view, the code that a virtual Gray-code projector fixed to the object's own frame would have painted on that surface point. The coarsest bit plane encodes front versus back, and it is resolved first, in the earliest and noisiest denoising steps.

The idea rests on a structural claim from structured light. The Janus problem is a fringe-order ambiguity: local appearance is plausible, but the global index of "which side am I on" is wrong or blended. The remedy that worked in metrology was a discrete, hierarchical global code resolved before fine detail. A continuous coordinate or depth map is not a substitute, and we make this a testable prediction rather than an assumption.

Core Claim

A multi-view diffusion model that jointly generates RGB and a hierarchical, discrete object-frame code will produce fewer duplicated fronts and more consistent geometry than one that generates depth or continuous coordinates. This holds when three conditions are met. First, the code is Gray-coded rather than continuous or purely periodic. Second, bit planes are tied to the model's coarse-to-fine schedule, with the most significant bits resolved in early high-noise steps, which govern layout according to Variational Control for Guidance in Diffusion Models. Third, the code lives in separate channels, never painted into the RGB image.

The reason for preferring discrete bits is simple. Suppose the model is unsure whether a pixel shows the front ($\theta = 0$) or the back ($\theta = \pi$) of the object, with probabilities $p$ and $1-p$. Here $\theta$ is the surface point's azimuth in the object frame. A regressor trained with squared error then predicts

$$\hat{\theta} = p \cdot 0 + (1-p)\cdot \pi,$$

a side-facing angle that corresponds to neither hypothesis. A predicted bit, by contrast, must commit to one side, and cross-view attention can then enforce a single consistent commitment.

Where This Sits

Three lines of work attack multi-view inconsistency, and this proposal borrows from and departs from each.

MVD-Fusion generates RGB and depth jointly and uses the noisy depth to reproject features across views. It supplies our method template of appending geometry channels. However, depth is camera-relative, and a two-faced head can have perfectly consistent depth. Depth says where surfaces are, not which side of the object they belong to.

MIRROR rectifies a drafted multi-view set with point trajectories computed from monocular depth. Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences mines correspondences from diffusion features. Both motivate this work: their correspondences come from passive cues, and the research file notes that these cues degrade on textureless or specular regions and can cement a hallucinated face rather than remove it. In our model, correspondence is generated as an explicit decodable signal rather than inferred afterwards.

Carve3D rewards round-trip reconstruction consistency measured with LPIPS, which mixes texture discrepancies with structural failures. Decoded codes offer a purely geometric consistency signal that we can report separately.

The structured-light literature supplies both the encoding and a design constraint. The Google Scanned Objects scanner pairs Gray codes with phase-shifted sinusoids for exactly this coarse-index/fine-refinement split. Xu et al. note that repetitive patterns cause periodic aliasing, which predicts that a sinusoid-only code should permit duplication.

The obvious alternative at this intersection is to paint virtual patterns into the RGB images, in the style of Active Stereo Without Pattern Projector. Two findings rule it out. That approach requires depth seeds, which a generator does not have. And Active-Passive SimStereo shows that image-guided refinement networks treat projected texture as real depth edges and inject artifacts. We therefore follow the logic of Textureless Deformable Object Tracking With Invisible Markers: the marker channel is separate, so appearance is untouched.

Scale-wise generators such as VAR show that coarse scales fix composition irrevocably. We use this as a constraint. A wrong front/back decision cannot be repaired later, so it must be made correctly, and checked, at the coarse stage.

Research Program

The code. For each surface point we compute object-frame azimuth $\theta$ and elevation $\phi$. We encode each angle as $K$ cyclic reflected Gray-code bits, cyclic so that $\theta$ wraps around cleanly, plus a sine/cosine phase pair at the finest period. The object frame is anchored as follows. For image-to-3D, $\theta=0$ is the direction toward the input camera. For text-to-3D, we use the canonical orientation of an orientation-filtered subset of the training meshes. Rendering the code on training meshes is trivial, because the "projector" is a lookup of surface coordinates.

The model. We fine-tune an existing multi-view latent diffusion backbone to denoise RGB latents plus code channels. The code channels are encoded at low resolution for the top bits and at full latent resolution for the fine phase. The loss on bit plane $k$ is weighted toward high-noise timesteps in proportion to its significance, so the front/back bit and quadrant bits are learned as early-step decisions.

The early check. At an intermediate denoising step we decode the most significant bits in every view. For each view $v$ with known camera, the decoded side label at pixel $x$ must agree with the side label obtained by reprojecting into view $v$ the points decoded in other views, using the current depth estimate. Views are sampled at the same object, so this is a well-posed cross-view test. We summarize it as

$$E_{\text{code}} = \frac{1}{|\Omega|}\sum_{(v,x)\in\Omega} \mathbb{1}\left[b^{(1..m)}_v(x) \neq b^{(1..m)}_{v'}(\pi_{v\to v'}(x))\right],$$

where $b^{(1..m)}$ are the top $m$ decoded bits, $\pi_{v\to v'}$ maps a pixel to its projection in another view $v'$, and $\Omega$ is the set of covisible pixel pairs. Samples whose $E_{\text{code}}$ exceeds a threshold are resampled from the coarse stage, borrowing the early-exit logic of multi-decoder hierarchical VQ-VAEs. The measured compute overhead is reported next to MIRROR's 2.6–2.9× inference time.

Preserving diversity. Carve3D reports that supervised fine-tuning on 3D data erodes appearance diversity. We therefore keep the RGB branch frozen apart from low-rank adapters and measure diversity explicitly.

Baselines

The naive reference is the same backbone fine-tuned on the same rendered data with the same number of steps, but without code channels. Any gain must exceed what the extra fine-tuning alone provides.

The strongest geometric competitor is MVD-Fusion. Its RGB-D design is the closest analog, and the file reports a Chamfer distance of 0.031 on Google Scanned Objects for it, against 0.043 for Point-E and 0.044 for Shape-E. We will retrain a depth-channel variant on our backbone and data so the comparison isolates code versus depth. We will also report the published number. Beating it means a lower Chamfer distance on the same GSO split, together with a lower duplicated-front rate.

The strongest training-free competitor is MIRROR. On T3Bench with VideoMV it reports PSNR rising from 21.67 to 24.82 and SSIM from 0.814 to 0.898. We apply MIRROR on top of our naive baseline with its released settings. We then also test MIRROR applied on top of our method, which checks whether the two are complementary.

For reward-based methods, Carve3D reports a Multi-view Reconstruction Consistency (MRC) score of 0.0606 against 0.0892 for Instant3D-10K on the DreamFusion 415-prompt set. We compute MRC for our models with the same reconstruction-model protocol. We also test a Carve3D-style fine-tune whose reward uses $E_{\text{code}}$ instead of LPIPS, holding the prompt set and number of updates fixed.

Four ablations isolate the key ingredient. The first replaces Gray bits with continuous $(\theta,\phi)$ regression. The second keeps only the sinusoidal phase with no Gray code, so fringe order is ambiguous. The third uses Gray bits without high-noise weighting. The fourth uses the full code but removes the early check.

All variants share data, resolution, number of views, compute budget and hyperparameter search budget.

Why not just…?

Why not just fine-tune on more multi-view 3D data? Carve3D found that 10× more supervised data (Instant3D-100K) did less for consistency than a consistency reward on 30 prompts. The CorrespondentDream paper shows that multi-view-trained models still hide pits and hollows under clean renders. Our naive baseline sits at our own data scale and measures this directly. If it closes most of the gap, the code idea fails.

Why not just generate depth, as MVD-Fusion does, or restrict attention to epipolar lines? Depth and epipolar lines constrain where a surface point lies, not which side of the object it is on. A consistently two-faced object satisfies both. MIRROR itself argues that epipolar lines leave points free to slide along the ray. The depth-channel baseline, retrained under identical conditions, tests whether "side identity" adds anything beyond geometry.

Why not just generate continuous canonical coordinates? This is the real threat, since it is the same idea without the structured-light encoding. The bimodal-averaging argument predicts that continuous regression produces side-facing blends and residual duplication where front/back is uncertain. The continuous-coordinate ablation settles this. If it matches the Gray-coded version, the contribution reduces to "generate object-frame coordinates," and we will report exactly that.

Why not just use monocular depth after drafting? That is MIRROR's route. Monocular depth reads each view alone and will model a hallucinated rear face as real relief. Running MIRROR on top of our naive baseline, and separately on top of our method, shows whether post-hoc rectification captures the same gain.

Proposed Experiments

Image-to-3D. We follow MVD-Fusion's protocol on Google Scanned Objects and reconstruct with NeuS. We report Chamfer distance, Volume IoU, and novel-view PSNR, SSIM and LPIPS.

Text-to-3D. We use T3Bench's 300 prompts, whose quality score smooths per-view scores over 161 icosahedron views specifically to penalize Janus artifacts; T3Bench reports that its automatic metrics correlate with human ratings at Spearman above 0.75. We also compute MRC on the DreamFusion 415-prompt set.

Direct Janus measurement. Because no file benchmark counts duplicated fronts directly, we add a blinded human annotation of front-feature duplication on the 100 animal and character prompts with the highest duplication rate under the naive baseline, following CorrespondentDream's paired-preference format.

Diversity. Diversity is measured as pairwise LPIPS across seeds per prompt, relative to the base model.

Decisive checks. Three results matter most. First, Gray code must beat continuous coordinates on duplication rate at equal Chamfer distance. Second, the sinusoid-only code should show more duplication than the full code, the aliasing prediction from structured light. If it does not, the fringe-order analogy is wrong. Third, $E_{\text{code}}$ measured at the early step must predict final-sample Janus annotations. If it does not, the early check is useless and the coarse-to-fine binding claim fails.