A 3D scanner pointed at a moving person or a flexible part sees only one side of it, at one moment, in one pose. If the shape looks unusual, there are three possible reasons. Part of it may be hidden from view. It may simply be bent differently than the reference shapes. Or it may be genuinely damaged: dented, torn, or carrying something stuck to it. Current 3D anomaly detectors cannot separate these cases. Their standard move is to rigidly align the scan to a stored template and flag whatever does not match. That works poorly even for rigid toys and cannot work at all for things that bend. We propose a detector that asks a different question of every observed point: can any plausible deformation of a normal shape explain this point? Points that no plausible shape can explain are defects. Points that a bent shape explains are normal. Regions with no points are occlusion and are never scored.
The method has three parts. It learns a compact code space of complete, deforming shapes. On top of that space it places an energy-based prior, which is a learned score that is low for codes of realistic shapes and high for distorted ones, not restricted to a simple bell-curve form. For each test scan, it then draws several possible complete shapes from the posterior given the visible points, using a robust likelihood that lets each point be declared an "outlier" instead of forcing the shape to bend toward it. The anomaly map is each point's average probability of being an outlier across these sampled completions. Hidden regions contribute nothing. Pose variation is absorbed by the prior. Defects remain unexplained because the prior assigns high energy to shapes that would reproduce them.
Core Claim
The claim is that three ingredients together remove the main false-positive sources of reconstruction-based 3D anomaly detection on non-rigid objects: an expressive latent energy prior over deformations, posterior sampling with an explicit per-point outlier variable, and scoring only observed points. The causes we target are occlusion boundaries, pose and articulation, and soft-tissue variation. Each ingredient answers a known failure. Deterministic completion collapses to a blurry mean of plausible shapes (PC2), so the residuals it produces mix defects with averaging error. A Gaussian latent prior over-generalizes and puts probability on distorted configurations (VAEBM), so it lets the fitted shape absorb a dent. This is the 3D version of the "identity shortcut" that defeats reconstruction detectors. Rigid registration cannot follow bending (Real3D-AD). The specific new element is the outlier responsibility computed under a latent-EBM posterior. Without it, the energy prior is just a better completion prior. With it, the prior decides how much of the scan has to be called anomalous.
Concretely, let $G(z)$ be the decoder that maps a latent code $z$ to a complete point set. Let $p_0(z)$ be a standard Gaussian, $E_\alpha(z)$ the learned energy, and $X=\{x_i\}$ the observed partial scan. We sample from
$$p(z \mid X) \propto p_0(z)\, e^{-E_\alpha(z)} \prod_i \Big[(1-\pi)\,\mathcal{N}\big(d(x_i, G(z));\, 0, \sigma^2\big) + \pi\, u\Big],$$
where $d(x_i, G(z))$ is the distance from observed point $x_i$ to the nearest point of the decoded shape, $\sigma$ is the sensor noise scale, $\pi$ is a small prior outlier rate, and $u$ is a uniform density over the scan volume. Each point's anomaly score is its outlier responsibility averaged over $K$ posterior samples $z^{(k)}$:
$$r_i = \frac{1}{K}\sum_{k=1}^{K} \frac{\pi u}{\pi u + (1-\pi)\,\mathcal{N}\big(d(x_i, G(z^{(k)})); 0, \sigma^2\big)}.$$
The object-level score is the mean of the top few percent of $r_i$. The distance is measured only from observed points to the shape, so missing regions cannot create residuals.
Research Program
Backbone. We use a topology-free point-cloud autoencoder: a point-set encoder and a folding-style decoder trained on complete normal shapes. This avoids the fixed mesh graph that limits Deformable Shape Completion With Graph Convolutional Autoencoders. That paper supplies our template for inference. It discards the encoder and optimizes a latent code to fit partial points under a Gaussian prior, and it reaches 2.51 cm mean error on synthetic DFAUST scans. We keep the latent-fitting idea and change three things: the prior, the likelihood (robust instead of squared error), and the output (posterior samples instead of one optimum).
Energy prior. The prior is trained as in Learning Latent Space Energy-Based Prior Model: an exponential tilting of a Gaussian, fit by maximum likelihood with short-run Langevin sampling. Langevin sampling is noisy gradient descent on the energy. That paper also motivates the approach, because its latent energy already flagged out-of-distribution digits on MNIST.
Posterior sampling. Posterior sampling borrows two pieces from Energy-Based Residual Latent Transport for Unsupervised Point Cloud Completion, which ran 8-step Langevin over latent residuals to complete static ShapeNet objects. We reuse its short-run residual parameterization and its stop-gradient trick. That work never asks whether an unusual shape is anomalous and never handles deformation.
Inference loop. We alternate Langevin steps on $z$ with closed-form updates of the outlier responsibilities, which is an EM-like scheme. Several chains start from encoder outputs of the partial scan.
Motion. Motion enters in two ways. First, the training set covers the full range of articulations and soft-tissue motion in DFAUST, so the prior encodes what normal non-rigid change looks like. Second, on sequences, the chains for frame $t$ start from the samples for frame $t-1$. This cuts the per-frame Langevin budget and lets a defect that persists across frames build up evidence over time. Earlier inference work warns that long chains can collapse into isolated modes (VAEBM), so we deliberately use many short chains and report how scores change with chain length.
Benchmark. No benchmark for this setting exists, so we build one. We hold out DFAUST subjects (Dynamic FAUST). We render single-view depth scans from varied cameras and inject parametric defects at random surface locations: dents, bulges, cuts with missing material, and attached foreign blobs, at several sizes. A second anomaly type covers deformation itself. These are joint bends past anatomical limits and local stretches that no subject produces. They test whether the energy prior separates valid motion from invalid motion, and not only damaged surfaces from clean ones. DHB sequences from IDEA-Net serve as a cross-capture test with the same defect injection. Real3D-AD serves as a rigid sanity check on real, high-precision scans with real defects.
Baselines
Naive references. The simplest reference is a deterministic point autoencoder trained on the same complete normal shapes. We encode the partial scan, decode one completion, and score each observed point by its distance to that completion. This is the "do nothing beyond reconstruction" baseline. It shows how much blur and pose mismatch cost. A second naive reference scores each point by its distance to the nearest training shape after rigid alignment.
Rigid registration. The strongest existing system is Reg3D-AD from Real3D-AD. It aligns partial scans to complete prototypes with RANSAC and scores points against a memory bank of PointMAE features. It is the only published method built specifically for partial-scan-versus-complete-prototype detection. It reports an average object-level AUROC of 0.704 on Real3D-AD, and most other methods there fall below 0.5. On Real3D-AD we run it with its released protocol. On the deformable benchmark we give it the same normal training shapes as its prototype bank. Beating it on Real3D-AD by any margin, while far exceeding it on deforming scans, would show that our method does not trade away rigid performance.
Template-based non-rigid fitting. This is the competitor most likely to be strong on humans. We fit the SMPL body model with per-vertex offsets to the partial scan, following the registration approach of Dynamic FAUST, and score points by their residual to the fitted surface. It uses a hand-built shape model our method does not have, so we treat it as a near-oracle. To beat it, our method must match its point-level average precision on clean-pose scans and exceed it on soft-tissue-heavy sequences and on large defects. In those cases the offsets either absorb the defect or are regularized so strongly that normal flesh motion shows up as residual.
Gaussian latent prior. Here we run the Litany-style method (Deformable Shape Completion With Graph Convolutional Autoencoders) on our topology-free backbone with the same robust likelihood and the same number of sampling steps. On synthetic DFAUST scans, its completion error is the reported 2.51 cm.
Static energy completion. We adapt Energy-Based Residual Latent Transport (paper) and score by its sample-variance uncertainty maps. This tests whether existing energy-based completion already does the job.
Ablations. Four ablations isolate the key ingredients. Gaussian prior versus energy prior, holding everything else fixed, is the central test. Squared-error versus robust likelihood tests the outlier variable. One posterior sample versus $K$ samples tests multi-hypothesis completion. Sampled completions from a conditional VAE versus energy-prior samples tests the sampler. All learned methods share one encoder and decoder architecture, the same training shapes, and an equal hyperparameter-search budget, with thresholds tuned on a held-out validation subject. Fixing the Langevin step count across conditions keeps compute matched.
Why not just…?
Why not just fit a Gaussian-prior VAE with a robust loss? This is the most serious threat. If an outlier-tolerant fit under a Gaussian prior already refuses to explain dents, the energy prior adds cost and no benefit. The case for the energy prior rests on evidence that Gaussian-prior VAEs place probability on non-data configurations (VAEBM) and that tilting the prior sharply improves sample quality: CelebA FID drops from 65.75 to 37.87 in Learning Latent Space Energy-Based Prior Model. That is image evidence, not 3D evidence. The prior ablation settles the question directly. If point-level average precision does not improve meaningfully on large defects and on the deformation-anomaly split, we will report that the outlier variable carries the method and drop the energy prior.
Why not just fit SMPL and threshold the residual? For humans this may well work, and we include it for that reason. It has two limits. It depends on a category-specific hand-built model that does not exist for cloth, organs, or industrial flexible parts. And its correction offsets must be either loose enough to absorb defects or tight enough to flag normal soft-tissue motion. The soft-tissue stratification and the DHB transfer test measure exactly this trade-off. If SMPL wins everywhere on humans, the contribution narrows to template-free categories, and we would say so.
Why not just rigidly align to a prototype, as Reg3D-AD does? Rigid alignment cannot represent bending. Even on rigid objects, Real3D-AD reports that partial views and truncated edges push many methods below chance. We expect it to fail on the deformable benchmark. The informative experiment is the rigid one: can a learned prior match registration when there is no deformation to model?
Why not just use ensembles or a conditional VAE for uncertainty? Sample spread from these models measures ambiguity in hidden regions. Our scoring never touches hidden regions, so what matters is whether the samples stay on the normal shape manifold while fitting the observed points. The sampler ablation tests this directly.
Proposed Experiments
Primary. The primary experiment is the DFAUST deformable-defect benchmark with held-out subjects. We report object AUROC and point-level AUROC and average precision, stratified by occlusion fraction, pose extremity relative to training poses, and defect size. No reported result was found on Lacuna for anomaly detection on DFAUST, so every number there is new and depends on our splits, which we will release.
Decisive checks. Three checks decide the proposal.
The first is the false-positive rate on clean scans in two places: points within a few centimeters of occlusion boundaries, and frames at extreme articulation. Our method should hold these near the rate seen on frontal neutral poses, while the deterministic baseline and Reg3D-AD rise sharply.
The second is the energy-versus-Gaussian ablation on defects large enough that a Gaussian-prior fit can bend toward them. This is where the claimed mechanism should show its largest gap.
The third is the deformation-anomaly split. Here a surface-residual method like SMPL fitting should fail, because the shape is locally smooth. The latent energy of the best-fitting code should still flag the scan.
Secondary. Completion quality on synthetic DFAUST scans is compared against the 2.51 cm Litany figure, to confirm that robust fitting does not degrade completion. On 3D-EPN we compare against the 9.4 Chamfer distance of Energy-Based Residual Latent Transport, to confirm the topology-free backbone is competitive on static shapes. On Real3D-AD we compare against Reg3D-AD's 0.704. We also report wall-clock time per frame, with and without warm starting across frames, because the roughly fourfold Langevin overhead reported for latent energy models (Learning Latent Space Energy-Based Prior Model) determines whether the method is practical on sequences.
References
- PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction
- VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
- Real3D-AD: A Dataset of Point Cloud Anomaly Detection
- Deformable Shape Completion With Graph Convolutional Autoencoders
- Learning Latent Space Energy-Based Prior Model
- Energy-Based Residual Latent Transport for Unsupervised Point Cloud Completion
- Dynamic FAUST: Registering Human Bodies in Motion
- IDEA-Net: Dynamic 3D Point Cloud Interpolation via Deep Embedding Alignment