Inverting Shape Generators from Unknown Poses

When we ask a pretrained generative model to reconstruct a real object, the procedure is called inversion: we search for the latent code that makes the generator reproduce the input. Every inversion method in the 2D and 3D literature quietly assumes that the input already sits in the generator's own coordinate frame. For photographs this assumption mostly holds. For 3D point clouds (scans, LiDAR crops, partial depth captures) it almost never does, because the object arrives rotated and shifted by an unknown amount. The obvious repair is to align the object first and invert second. But alignment needs a reference shape, and the right reference is the very instance that inversion is trying to recover. This is the same chicken-and-egg problem that makes classical registration fall into false solutions when started far from the truth.

We propose to solve pose and shape together. The pose is the rigid rotation and translation, and the shape is the generator's latent. The coarse-to-fine schedule for the joint solve comes from the noise levels of a diffusion model, which is a generator that turns random noise into samples by denoising it step by step. At high noise, the model's best guess of the clean shape is a blurred, category-typical form. Matching the observation against that blurred guess with a wide-bandwidth distribution distance gives a smooth objective with few false solutions over rotations. As noise decreases, the guess sharpens toward the specific instance, and the matching bandwidth shrinks with it. The claim is that tying the matching bandwidth to the diffusion noise level gives inversion a wide capture range (the set of starting poses from which the solver still converges to the right answer) without any separate registration stage. It should also preserve reconstruction fidelity at the end.

Where this sits

Two bodies of work meet here and neither covers the gap. On the inversion side, PointInverter maps point clouds into a sphere-guided GAN through a global encoder followed by local latent refinement. It assumes canonically aligned ShapeNet objects, and a rotated input sends its encoder to wrong latents. This is the most direct evidence for the gap. The 2D editing literature shows the same blind spot: TF-ICON explicitly lists viewpoint rigidity as a limitation of inversion-based composition. Several 2D works tune inversion trajectories against the generator's own numerical drift. ReNoise uses fixed-point renoising, Free Lunch for Stabilizing Rectified Flow Inversion uses proximal velocity averaging, and Timestep Rescheduling uses non-uniform timesteps. All three treat the latent as the only unknown, while we add a rigid pose that the latent cannot absorb. Exact solvers such as BELM, which reports latent MSE of 0.000 on COCO 2014, and BDIA supply the method for our final stage. Once pose is resolved, we use them so that the remaining reconstruction error cannot be blamed on the solver. Instability in Diffusion ODEs constrains the design. It shows that the noise-to-data map stretches space strongly, so small latent errors grow into large output errors. We therefore avoid running gradient descent on the initial noise through the full generation chain early on, and do the hard search at high noise levels where the map is still smooth.

On the registration side, Go-ICP finds globally optimal rigid alignment by branch-and-bound, a search that splits pose space into cells and discards cells whose error bound is too high. It needs a known model shape. Feature-Metric Registration widens the capture range by aligning learned features instead of coordinates, reporting accuracy up to 80° of initial rotation on ModelNet40. MMD-Reg aligns the two point sets as probability distributions using maximum mean discrepancy (MMD) under a Gaussian kernel. Annealing the kernel bandwidth widens the capture range. MMD-Reg supplies our matching objective and its annealing idea. Its annealing, however, smooths a fixed target. Our annealing smooths a target that is itself being generated, so the target moves from category prior to instance at the same rate the landscape sharpens. None of these registration methods can complete missing geometry or return an editable latent.

Core Claim

Let $Y$ be the observed points. Let $x_\sigma$ be a noisy shape latent at diffusion noise level $\sigma$, and let $D(x_\sigma,\sigma)$ be the denoiser's estimate of the clean point cloud. Let $(R,t)$ be a rotation and translation. At each level $\sigma_k$ of a decreasing schedule, we solve

$$ \min_{R,\,t,\,x_{\sigma_k}} \; \mathrm{MMD}_{\ell(\sigma_k)}\big(R\,D(x_{\sigma_k},\sigma_k)+t,\; Y\big) \;+\; \lambda\,\|x_{\sigma_k}-\tilde{x}_{\sigma_k}\|^2, $$

where $\mathrm{MMD}_\ell$ is maximum mean discrepancy with Gaussian kernel bandwidth $\ell$, and $\ell(\sigma)=\alpha\sigma+\ell_{\min}$ couples bandwidth to noise. The anchor $\tilde{x}_{\sigma_k}$ is the previous level's solution re-noised to $\sigma_k$, and $\lambda$ keeps the latent near the diffusion trajectory. The linear coupling has a simple rationale: under Gaussian corruption, the denoiser's output at noise $\sigma$ resolves detail only down to roughly scale $\sigma$, so a kernel narrower than that would match noise rather than structure.

At the coarsest level the shape estimate is nearly a category mean and the objective is low-dimensional and smooth. There we run a cheap global search over rotations, either branch-and-bound in the style of Go-ICP or a dense rotation grid, with the latent held at its prior mean. That search seeds local refinement. The final level hands the pose and latent to an exact BELM round trip, which yields an editable latent. We also make a falsifiable prediction. The success rate as a function of initial rotation angle should stay flat out to 180°, apart from shapes with true symmetries. Register-then-invert pipelines should instead degrade on partial and atypical shapes, where the category template is a poor reference.

Research Program

We first train a point cloud diffusion model on ShapeNet categories, since the file names no pretrained 3D diffusion model, and measure its exact-inversion fidelity on canonically posed inputs. This sets the ceiling. We then map the objective landscape empirically by sweeping rotations on a sphere at each noise level and counting local minima. The goal is to test whether the coupled bandwidth actually removes false solutions, and whether it does so better than a fixed bandwidth or the noise level alone. Next comes the full solver, followed by a study of its failure modes. The main ones we expect are partial scans below 40% coverage, where the file notes that MMD suffers from mass-mismatch bias, and near-symmetric categories, where we report symmetry-aware rotation error. We will try a partial-observation variant that matches only the observed side, by weighting generated points by their visibility under the current pose. Finally, we check that the recovered latent supports the part edits PointInverter demonstrates.

Baselines

The naive reference is PointInverter applied directly to rotated inputs, with no pose handling. It shows how much is lost by ignoring pose. We run it with its released pipeline on the same perturbed ShapeNet test sets. Beating it is expected, so it only calibrates the size of the problem.

The strongest competitor is register-then-invert. We align the input to the category mean shape with Go-ICP, which is globally optimal for a known model, or with Neural MMD-Reg or FMR, and then run our own diffusion inversion with the exact BELM solver. This is the reviewer's default and genuinely strong. Go-ICP reports a 100% success rate in finding the global minimum on Stanford Bunny across 100 random initializations, and FMR reports accurate registration up to 80° on ModelNet40. Those numbers concern alignment to a known shape, not to a category template, and we found no reported result on Lacuna for either method used to align instances to a category template before inversion. For a fair comparison, the inverter is identical across methods, and each registration method gets the same wall-clock budget as our full solver. Our method beats this baseline if it shows higher success rates (rotation error under 5°, translation under 5% of object scale) and lower Chamfer and earth mover's distance at matched compute, specifically on partial and atypical shapes. Roughly matching on complete typical shapes would be acceptable.

A multi-start baseline runs plain joint pose-and-latent gradient descent at the clean noise level from many random rotations, with the restart count chosen so total generator evaluations match ours. It tests whether our gain is just more compute. A rotation-augmented baseline retrains PointInverter's encoder with random rotations, testing whether an amortized encoder learns pose on its own.

Three ablations isolate the key ingredient. The first uses a fixed bandwidth with noise annealing alone. The second anneals bandwidth with a fixed category-mean target, which is essentially MMD-Reg against a template. The third removes the coarse global search, and a fourth variant replaces MMD with Chamfer distance. If the second ablation matches the full method, the generative prior is not earning its place and the claim fails.

Why not just…?

Why not just register to a template first, then invert? Our main experiment tests exactly this. Our reason to expect a gap is that a category mean is a biased reference: a long-necked chair or a cropped car scan has a different best-fit pose against the mean than against itself. Even an optimal registration method such as Go-ICP is then optimal for the wrong target. If register-then-invert matches us on partial and atypical subsets, we will report that the joint formulation is unnecessary for these categories.

Why not just train a rotation-invariant or canonicalizing encoder? Pose-invariant feature separation in the style of Zhang et al. is a real threat on complete shapes. Its weak point is partial input, because a canonical frame estimated from an incomplete surface is ambiguous. We include an equivariant canonicalizer followed by inversion as an additional baseline and report results stratified by coverage, so the outcome is decided by data and not by argument.

Why not just use many random restarts? Restarts recover the global solution only if one start lands in the correct basin, and in 6-DoF pose plus a high-dimensional latent that requires many starts. The compute-matched multi-start baseline settles the question. If it closes the gap, our contribution reduces to efficiency, and we will say so.

Proposed Experiments

We test on ShapeNet categories (chair, airplane, car), with each test shape randomly rotated across the full sphere and translated. Each shape appears in three conditions: complete, with Gaussian noise, and cropped to 30–70% partial views. We also test ModelNet40 shapes from categories absent from training to probe how the generative prior behaves out of distribution, where the FMR and MMD-Reg numbers give context. The decisive figure plots success rate against initial rotation angle from 0° to 180° for every method, separately for each coverage level. The second decisive check is reconstruction fidelity, measured as Chamfer and earth mover's distance against the full ground-truth shape. This matters because recovering unobserved geometry is what the generator adds over registration. The third check is editability: we apply PointInverter-style part edits to recovered latents and measure how far the unedited regions move. We will count the proposal a success if the curve stays flat to 180° on non-symmetric categories and beats register-then-invert at coverage below 50%. We will count it a failure if the template-annealing ablation matches the full method.