To work out how a drawer slides or a laptop lid swings from a sequence of 3D scans, a system has to decide which points in one frame correspond to which points in the next. Most learned pipelines for this assume each point has exactly one partner. They either force the matching to be a near-permutation or let every point pick a nearest neighbor. Both assumptions break on articulated objects. Opening a drawer reveals its interior, a closing lid hides the surface behind it, and a depth sensor samples the same surface patch more densely when it faces the camera. Machine translation ran into this problem decades ago. Forcing every word to align to exactly one word fails on idioms, on agglutinative morphology, and on words with no counterpart. That field's answer was unbalanced transport: an assignment in which "mass" can be destroyed or created at a price, so that some elements are allowed to have no partner.
We propose to bring that relaxation into articulation estimation and to tie it to the physics. Each moving part is constrained to move along a single screw: a rotation about an axis plus a translation along that same axis. By Chasles' theorem this one form covers hinges, sliders and helical joints without first classifying the joint type. The screw motion predicts where points should land. Unbalanced transport decides which points actually have a partner. Each part is given its own mass budget, so a thin handle cannot be absorbed by the large cabinet body. The result is an alternating optimizer: correspondence informs the screw, and the screw informs correspondence. We expect it to hold up under disocclusion, uneven sampling and small moving parts, which are the cases where current one-to-one and nearest-neighbor methods fail.
Core Claim
Articulation estimates from point cloud sequences fail mostly because of the correspondence model, not the motion model. Our hypothesis is that coupling three ingredients removes most of those failures, and that removing any one of them brings the failures back. The first is unbalanced optimal transport, which allows unmatched points. The second is a per-part source marginal, which gives each part equal total mass. The third is a single screw axis per joint, shared across all frames, with only the joint displacement changing over time.
Let $x_i$ be points of a reference frame with soft part assignments $s_{ik}$ and $y_j^t$ points of frame $t$. Each part $k$ has a unit twist $\xi_k$, meaning screw axis direction, pivot and pitch. The twist is shared across time, while a scalar displacement $\theta_k^t$ changes per frame. The rigid motion is $g_k^t=\exp(\theta_k^t \hat\xi_k)$. For each frame we solve
$$\min_{T^t \ge 0} \sum_{i,j} T^t_{ij}\, \Big\| y^t_j - \textstyle\sum_k s_{ik}\, g_k^t x_i \Big\|^2 + \lambda_1 \mathrm{KL}(T^t\mathbf{1}\,\|\,p) + \lambda_2 \mathrm{KL}(T^{t\top}\mathbf{1}\,\|\,q) + \varepsilon\,\Omega(T^t).$$
Here $T^t$ is the transport plan, $\Omega$ is the entropy regularizer, and $q$ is uniform over target points. The source marginal is not uniform: $p_i=\sum_k s_{ik}/(K\,m_k)$, where $m_k=\sum_i s_{ik}$ and $K$ is the number of parts. Each part therefore owns $1/K$ of the mass regardless of how many points it has. The first KL term lets disoccluded or hidden points go unmatched. The part-balanced $p$ makes ignoring a small part as costly as ignoring the base.
Given the plans, the twists and displacements are refit by weighted least squares on the screw manifold, pooled over all frames. This pooling enforces a true one-degree-of-freedom joint rather than a free-floating 6-DOF motion. The cost and marginal form are taken directly from Unbalanced Optimal Transport for Unbalanced Word Alignment. The new elements are the part-balanced marginal and the screw-predicted cost.
Research Program
The work builds on four groups of prior work, which play different roles.
The motivation comes from the articulation side. P^3-Net tracks points across frames with a Gumbel-Sinkhorn layer that drives matching toward a near-permutation. It then predicts joint attributes with separate revolute and prismatic heads. It is exactly the one-to-one bottleneck we target, and it does not use a continuous joint model. OP-Align registers parts with a density-aware Chamfer distance, a nearest-neighbor loss, and reports collapse on small moving parts such as thin handles. That is the scale-imbalance failure our part-balanced marginal is meant to fix. Multi-body SE(3) Equivariance discovers rigid parts without labels. However, it predicts an unconstrained rotation and translation per part from soft feature matching with no handling of unmatched points.
The motion model comes from screw-based work. ScrewSplat and Category-Independent Articulated Object Tracking with Factor Graphs show that continuous twists avoid joint-type classification and generalize across joint types. ScrewSplat works from photometric rendering, and the factor-graph tracker works from clustered motion maps. Neither has an explicit correspondence model that tolerates missing partners.
The transport machinery comes from two sources. OTAlign supplies the unbalanced objective. RobOT shows that unbalanced transport over 100k points runs in under a second with symbolic tensors, using O(N+M) memory, which makes per-frame transport affordable. RobOT, however, models global rigid or smooth deformable motion, never multibody joints.
The linguistic lens also sets a design constraint. BinaryAlign shows that softmax normalization dilutes probability when one element has several partners. Row-softmax soft matching of the kind used in multibody segmentation has the same problem. This is why we use transport plans with unbalanced marginals instead of attention rows.
We will build the method in three stages.
- Test-time optimizer, single joint. We start with an optimizer over fixed per-point features: geometry plus features from the public Multibody_SE3 codebase. This isolates the correspondence question from representation learning.
- Multiple independent joints. We extend to several parts, each attached to the base by its own joint. The soft assignments $s_{ik}$ are updated in the same alternating loop. Part counts are over-provisioned, and empty parts are pruned when their transported mass falls to the null level. This is a mass-based alternative to ScrewSplat's parsimony weight, which the file reports as brittle.
- Learned features. Only after stages 1 and 2 work do we train the feature extractor through the differentiable transport layer. We leave nested kinematic chains out of scope.
Baselines
The naive reference fits screws to nearest-neighbor correspondences. Concretely, it is iterative closest point inside each part followed by the same pooled screw fit. It uses the same part initialization, the same features and the same iteration budget as our method. It tells us how much any transport model buys at all.
The strongest learned one-to-one competitor is P^3-Net. On its Motion Dataset it reports an end-point error of 0.181 and a segmentation IoU of 0.84 on PartNet (P^3-Net). We will retrain it with its released settings on our training splits. We will also report it on its own benchmark so that our numbers can be checked against the published ones.
The strongest unsupervised multibody method is Multi-body SE(3). It reports 63.8% average precision for segmentation on mbs-sapien (Multi-body SE(3) Equivariance). We run its code unchanged and attach the same post-hoc screw fit to its per-part motions so that joint errors are comparable. To beat it, we must match or exceed that segmentation AP on unperturbed mbs-sapien and clearly reduce joint-axis error under our occlusion protocol.
For small-part alignment, OP-Align reports a joint direction error of 6.63° and a part IoU of 80.70 on its synthetic benchmarks (OP-Align). It is evaluated there with its required category priors. We make no claim on its full benchmark beyond the small-part subset.
The most dangerous competitor is a two-stage pipeline built from D-RobOT. It computes unbalanced-transport scene flow between frames, clusters the flow into rigid parts and fits screws afterward. Its published 2.23 cm end-point error is on KITTI scene flow (RobOT). We found no reported result on Lacuna for D-RobOT on articulated-object benchmarks, so we will run it ourselves. It gets the same point budget and the same transport hyperparameter search (λ, ε) as our method.
Three ablations each remove one ingredient while keeping everything else, including tuning budget:
- balanced Sinkhorn transport in place of unbalanced transport;
- unbalanced transport with a uniform source marginal in place of the part-balanced one;
- a free SE(3) motion per frame in place of the shared screw axis.
Beating a baseline means a lower joint-axis angle and pivot error, averaged over at least three seeds. It also means non-overlapping confidence intervals on the perturbed splits.
Why not just…?
Why not just run robust unbalanced scene flow and fit screws afterward? This is the real threat. If D-RobOT flow followed by clustering and screw fitting matches our accuracy, the coupling is not worth it, and the contribution shrinks to "unbalanced transport helps articulation." We expect coupling to matter for two reasons. First, a screw shared over many frames rejects flow vectors inconsistent with a single axis. Second, the screw-predicted positions tell the transport where unmatched mass should be: at newly revealed surfaces, not scattered everywhere. The decisive experiment compares the two-stage pipeline with ours as the disocclusion fraction grows. If the curves coincide, we will report that finding and drop the coupling claim.
Why not just use balanced Sinkhorn, or nearest-neighbor Chamfer, as current pipelines do? Prior evidence points against both. Balanced transport must match every point, which OTAlign shows fails on alignment data with up to 63.8% unaligned tokens (MSR-RTE). RobOT shows the same failure under partial overlap in 3D. Chamfer-style losses are what OP-Align reports collapsing on small parts. Our balanced-transport and naive ablations test both claims directly in the articulated setting rather than borrowing them from other domains.
Why not just reweight the Chamfer loss per part instead of using transport? Per-part reweighting does address scale imbalance, and it is cheap. It does nothing about disocclusion, though: a revealed drawer interior still gets pulled toward the nearest old surface. We will include a per-part-weighted Chamfer variant. We expect it to close the small-part gap but not the occlusion gap. If it closes both, the transport machinery is unnecessary for this problem, and we would say so.
Proposed Experiments
We generate single-view depth sequences of PartNet-Mobility objects in SAPIEN. Single-view rendering makes disocclusion and view-dependent sampling density arise naturally rather than being injected as synthetic noise. We stratify test sequences by the fraction of points with no true partner in the next frame, which the simulator labels exactly. We also tag a small-part subset: objects whose moving part has under 5% of the points, such as handles, levers and lids on small boxes. In addition, we evaluate on the unperturbed mbs-sapien test set and the P^3-Net Motion Dataset, so that published numbers anchor our reimplementations.
The metrics are:
- joint-axis angle error, pivot distance and joint displacement error;
- part segmentation IoU and AP;
- end-point error of predicted correspondences;
- calibration of null detection: whether points the plan leaves unmatched are truly the disoccluded ones, scored by ROC AUC against simulator labels, in the spirit of OTTAWA's evaluation of null routing in OTTAWA.
We will run three decisive checks.
- Disocclusion sweep. Joint error for P^3-Net and balanced transport should rise with the unmatched-point fraction, while our method stays roughly flat. If ours rises just as fast, the central hypothesis is wrong.
- Small-part subset. Removing the part-balanced marginal should bring back the collapse OP-Align describes, with moving-part IoU dropping toward zero, while the full method keeps the small part.
- Joint-type transfer. Following the factor-graph tracker's protocol, we fit sliders only and test hinges. This confirms that the screw form avoids the categorical bias reported there for ScrewNet, and checks that the transport relaxation does not reintroduce it.
References
- Unbalanced Optimal Transport for Unbalanced Word Alignment
- P^3-Net: Part Mobility Parsing from Point Cloud Sequences via Learning Explicit Point Correspondence
- OP-Align: Object-Level and Part-Level Alignment for Self-supervised Category-Level Articulated Object Pose Estimation
- Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation
- ScrewSplat: An End-to-End Method for Articulated Object Recognition
- Category-Independent Articulated Object Tracking with Factor Graphs
- Accurate Point Cloud Registration with Robust Optimal Transport
- BinaryAlign: Word Alignment as Binary Sequence Labeling
- OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection