Stage-Amortized Global Context for Unrolled Point Cloud Denoising

Cleaning a noisy 3D scan is an iterative job. The best learned point cloud denoisers move each point a little at a time toward the true surface, using a network that looks at the point's neighborhood. Each step is cheap only if the network sees a small neighborhood. Seeing the whole shape would help: it lets the model notice that a thin chair leg resembles three others, or that a flat panel continues past an occlusion. But paying for whole-shape context at every one of $K$ steps is what makes iterative methods slow and memory-hungry. Our observation is that points barely move between consecutive steps, so their neighbor lists, their ordering along a space-filling curve, and any whole-shape summary computed from them go stale slowly. We propose to compute these expensive quantities once, carry them across many cheap local steps, and refresh them only rarely. The goal is global context at close to the price of a purely local iterative denoiser.

We would build this as a deep unfolded network, meaning an iterative optimization algorithm whose steps become trainable network layers trained end to end. It would act directly on point coordinates, and its stages would share weights so that one trained model can run for whatever number of steps the hardware budget allows. The result targets three problems together. Unfolded networks have cost that grows linearly with depth and a depth that cannot change after training. Local operators have a limited receptive field. Point clouds are irregular, so their neighborhoods have to be recomputed as points move.

Core Claim

Inside an unrolled coordinate-denoising loop, the global, permutation-sensitive machinery can be refreshed every $m$ stages instead of every stage, with negligible loss in accuracy. That machinery consists of the $k$-nearest-neighbor graph, the Hilbert-curve serialization that orders points along a space-filling curve, and a linear-time state space model (a recurrent sequence model whose cost grows linearly with length) that summarizes the whole cloud. If the claim holds, global context becomes affordable at every depth. It would also settle an argument in the literature. Score-Based Point Cloud Denoising argues that score networks must stay local to learn reusable primitives such as planes and edges. Non-Local Part-Aware Point Cloud Denoising argues that locality causes over-smoothing and blindness to repeated structure. Our design keeps the per-step update local and lets slowly refreshed global context modulate it, so the two positions can be tested against each other under a controlled cost.

One stage updates the noisy coordinates $X^{(k-1)} \in \mathbb{R}^{N\times 3}$ against the observed scan $Y$ (point $i$ of $X$ starts at point $i$ of $Y$, so index correspondence is fixed and the update is permutation-equivariant):

$$X^{(k)} = X^{(k-1)} + \eta_k \Big[\, s_\theta\big(X^{(k-1)};\ \mathcal{N}^{(\lfloor k/m \rfloor)},\ c^{(\lfloor k/m \rfloor)}\big) - \lambda_k \big(X^{(k-1)} - Y\big) \Big].$$

Here $s_\theta$ is a local network that predicts a per-point step toward the surface, and $\mathcal{N}$ is the neighbor graph. The term $c$ holds per-point global context features from a state space model run over the Hilbert-serialized cloud. Both $\mathcal{N}$ and $c$ are recomputed only every $m$ stages. The scalars $\eta_k$ and $\lambda_k$ are a learned step size and a learned data-consistency weight; $\lambda_k$ pulls points back toward the measurement. The data-consistency term is the piece missing from heuristic gradient ascent. We expect it to counter the drift and volume shrinkage that the denoising literature documents. On real LiDAR, a variant restricts each point to move only along its sensor ray, which is the physically dominant direction of range noise.

Research Program

The work has three parts.

Unrolled coordinate denoiser. We first build the unrolled denoiser for coordinates. The closest unrolling work on point clouds, Graph Signal Restoration Using Nested Deep Algorithm Unrolling, unrolls ADMM with graph-independent learned weights. It restores only color signals on a fixed, noise-free graph, so its graph never changes. Ours changes at every stage, and that is the whole difficulty. We take the idea of a single conditioned network that serves many operating points from ISTA-Net++, which modulates shared stages by sampling ratio. Here we condition on stage index and noise level. Weights are tied across stages, and training samples $K$ at random from a range, so that inference can stop at any $K$. To keep training memory flat as depth grows, we use gradient checkpointing per stage, with reversible stage computation as in RUN as a fallback.

Global context module. We reuse Hilbert serialization with dual traversals from PointMamba. That work showed linear-time global modeling on point clouds, but only as a single-pass classifier. 3DMambaComplete extends the same idea to completion, again feedforward. The context features are injected into $s_\theta$ through scale-and-shift modulation, so the update stays local in form.

Staleness analysis. We measure directly how quickly the neighbor graph, the serialization order and the context features drift across stages: the fraction of $k$-NN memberships that change per stage, and the rank displacement along the Hilbert curve. This turns the refresh interval $m$ from a guess into a quantity read off data, and it predicts where amortization should break, namely at high noise in early stages when points move far. That prediction motivates an adaptive schedule: refresh when the mean displacement since the last refresh exceeds a fraction of the median neighbor spacing.

Baselines

The naive reference is the noisy input itself, together with classical graph Laplacian regularization run to convergence. The research file gives no reported result for either on PU-Net, so we will compute both.

The strongest direct competitor is score-based denoising (Score-Based Point Cloud Denoising). It trains a DGCNN to estimate the gradient toward the clean surface and runs hand-tuned gradient ascent, rebuilding neighbors at every step. It is the most relevant baseline because it is already iterative and local; our method differs in end-to-end training through the unrolled steps, the data-consistency term, and amortized global context. That paper reports a Chamfer Distance of 4.708 on PU-Net at 3% Gaussian noise, against 13.067 for the single-pass PCNet. We will rerun its released configuration on identical splits and noise samples, and match its total inference compute. Beating it means lower Chamfer Distance at equal wall-clock time, or equal Chamfer Distance at clearly lower latency and peak memory.

Two further competitors cover the other directions. The first is the manifold-reconstruction autoencoder Differentiable Manifold Reconstruction for Point Cloud Denoising, which reports a point-to-surface error of 2.32 on ModelNet40 at 3% noise against 3.28 for the next best method. We include it because it is the strongest single-pass method in the file. The second is the non-local denoiser of Non-Local Part-Aware Point Cloud Denoising, which reports only relative gains over TotalDenoising: 48.1% in Chamfer and 44.1% in Hausdorff distance at 2% noise. We include it because it represents global context bought with quadratic feature-space search. We run it without its part-label loss, so that the supervision available to every method is the same.

The ablations isolate the key ingredient. The local-only version is our unrolled network with $c$ removed. The full-refresh version recomputes $\mathcal{N}$ and $c$ every stage ($m=1$); it is the accuracy ceiling for amortization. The no-consistency version sets $\lambda_k=0$. All learned variants receive the same training data, the same number of training iterations and the same hyperparameter search budget.

Why not just…?

Why not just run score-based denoising with its existing loop and tune the number of steps? This is the most serious threat, because the local-is-enough argument of Score-Based Point Cloud Denoising may simply be right for small, isotropic noise. Our local-only ablation trains the same kind of loop end to end with data consistency. If that ablation alone matches the full model, the contribution reduces to "unroll and train end to end," and we will report it that way. We expect global context to matter mainly at 2–3% noise and on shapes with repeated thin parts, and we test exactly those slices.

Why not just use a single-pass global network such as a PointMamba-style encoder–decoder that regresses displacements? The file documents that single-pass displacement regression shrinks shapes: PCNet's 13.067 versus 4.708 for iterative refinement on PU-Net. We will still train a feedforward PointMamba displacement regressor at matched parameters and FLOPs. If it ties our model, the iterative structure is not needed.

Why not just recompute everything every stage and accept the cost? Because cost is the point of the proposal: each stage pays neighbor search and a full-cloud scan, and those per-stage costs add up over all $K$ stages. The full-refresh ablation measures what that buys. Amortization is justified only if its accuracy gap to full refresh is small while latency and memory fall substantially at 50k+ points.

Why not just voxelize or project onto images and use grid-based unfolding? FoldDiff documents the cubic cost and quantization artifacts of voxels and the topological tearing of 2D folding, and grid mechanisms such as CCNet's criss-cross attention need row and column axes that point sets lack. We consider this alternative weaker and do not plan to run it.

Proposed Experiments

We train on the PU-Net training split with Gaussian noise at 1–3% of shape scale. We test on the PU-Net and PointCleanNet test sets, and on ModelNet40 at 3% noise for comparison with the manifold-reconstruction method. Metrics are Chamfer Distance and point-to-surface error, plus latency and peak GPU memory measured at 10k, 50k and 100k points. Generalization is tested on Laplace noise, which is absent from training, and qualitatively on the real LiDAR of Paris-rue-Madame; the file reports no numbers for that dataset, and we will use the ray-constrained variant there.

Three checks decide the outcome.

Staleness curve. We plot accuracy against refresh interval $m$ at fixed $K$. The claim survives if $m=4$ stays within a small margin of $m=1$ while cutting the per-depth cost of neighbor search and scanning by about the factor $m$. The claim fails if the gap opens even at small $m$.

Anytime-depth curve. We evaluate one trained model at $K$ from 2 up to twice the training maximum. Understanding approximate and unrolled dictionary learning for pattern recovery found that excessive unrolled depth causes instability. Accordingly, we test whether error plateaus or diverges beyond the trained range. If it diverges, we add Lipschitz constraints on $s_\theta$ in the manner of Prompting Lipschitz-constrained network for multiple-in-one sparse-view CT reconstruction.

Local versus global split. We compare the local-only and full models on cross-category generalization, training on half of the categories and testing on the rest. This tests directly whether global context helps, as the non-local view predicts, or causes overfitting, as the local-score view predicts.