Trackers for deforming surfaces (a waving shirt, a bending arm, a hand squeezing a toy) usually model the surface as a sparse graph of small patches, each allowed to move almost rigidly. This works until the surface is smooth. On a flat torso or a cylindrical forearm, the depth data cannot tell a patch that stayed put from one that slid sideways along the surface, so the estimated surface drifts tangentially over time. Dynamic FAUST showed that geometry-only registration slides in exactly this way on human bodies. It fixed the problem by painting high-contrast patterns onto skin. Real subjects do not come painted.
We propose that sliding is a specific, predictable error: it lives in the directions that local geometry leaves unconstrained. Color information should therefore be fused into the tracker only along those directions, and trained with a loss that catches drift accumulating over several frames rather than per frame pair. In plain terms, each patch asks the color image "which way did I slide?" only along the axes where its own shape cannot answer. A multi-frame consistency loss teaches it when to trust the answer. The prediction is that gains concentrate in the tangential error component, in low-curvature regions, and at long time horizons. That signature can be checked directly.
Core Claim
Several terms need a short definition. A deformation graph is a set of nodes sampled over the surface, each carrying a small rigid motion. As-rigid-as-possible (ARAP) regularization penalizes neighboring nodes whose motions disagree. Cross-modal attention is a transformer operation in which features from one sensor (here depth geometry) form queries that retrieve features from another sensor (here color).
The claim is that a tracker which restricts color-derived corrections to the geometric null space of each node, and learns when to apply them from a truncated multi-frame cycle loss, reduces long-horizon tangential drift more than any of three alternatives. Those alternatives are unrestricted fusion (concatenation or gated attention), a photometric term in the energy, or post-hoc smoothing.
For node $i$ with neighborhood surface normals $n_j$, let $M_i=\sum_j n_j n_j^\top$, with eigenpairs $(\lambda_k, e_k)$. Small $\lambda_k$ marks a direction that point-to-plane alignment cannot constrain; on a plane, these are the two tangent directions. We form a soft projector $P_i=\sum_k \phi(\lambda_k/\lambda_{\max}) e_k e_k^\top$, where $\phi$ is a fixed decreasing function. The fused node displacement is
$$d_i = d_i^{\text{geo}} + g_i\, P_i\,(a_i - d_i^{\text{geo}}).$$
Here $d_i^{\text{geo}}$ is the displacement implied by geometric correspondences and $a_i$ is the displacement implied by cross-modal attention. The gate $g_i\in[0,1]$ is predicted from that attention's own confidence. Where geometry is informative, $P_i\approx 0$ and color cannot override it. Where geometry is degenerate, color supplies exactly the missing components.
The novelty lies in this geometry-derived projection and in how the temporal loss trains it. It does not lie in "RGB-D plus attention plus ARAP," which the file already covers piecemeal.
Relation to Prior Work
Two pieces of prior work supply the method we build on.
- Neural Non-Rigid Tracking places a differentiable Gauss-Newton ARAP solver after a convolutional correspondence network with learned confidence weights. We keep its solver and training-through-the-solver design. We replace its static correspondence stage with the null-space fusion above. We add a multi-frame loss; its training is pairwise against keyframes, which the file notes causes drift.
- Gated cross-attention for drug-target interaction supplies the learnable gate and sparse attention that prevent one input from swamping the other. Our gate is additionally shaped by a geometric projector rather than learned freely.
Several works motivate the problem.
- Dynamic FAUST provides the evidence that geometry alone slides.
- DeepDeform shows that joint color-and-geometry matching beats geometry-only matching. It stacks the channels at the input and uses no mechanism that decides where color should matter.
Other works constrain the design.
- Video Depth Anything argues that optical-flow warping losses are expensive and break at occlusions. Its temporal gradient matching avoids flow but acts on 2.5D depth values. It says nothing about where surface points go tangentially, which is exactly the component we care about.
- MS-TCN shows that truncating a temporal penalty keeps real transitions from being smoothed away. We borrow truncation so that tears, fast motions and occlusions do not dominate the loss.
The pairwise non-rigid matchers are adjacent but answer a different question. GraphSCNet injects local distance-preservation scores into self-attention to prune bad correspondences. Lepard uses relative 3D position encodings inside cross-attention between two point clouds. Both are single-pair and geometry-only. Neither can create tangential information that the geometry lacks.
The closest three-way combination is Depth-Guided Metric-Aware Temporal Consistency. It fuses RGB and depth with gated cross-attention and enforces bone-length consistency over time, but only inside the SMPL human body model. We target arbitrary surfaces without a template. 3D Implicit Transporter discovers temporally stable keypoints through cross-frame attention and occupancy reconstruction. It has no explicit local rigidity, and its keypoints can fall inside the object's volume. Our quantities are surface-bound graph nodes by construction.
Research Program
Architecture. Depth points around each graph node are encoded into node tokens. Each token queries color features from a local window in the target frame, centered on the node's geometric prediction. The window is a masked neighborhood, not all pixels, which both limits the quadratic cost of cross-attention and avoids the feature dilution the file flags for unmasked attention. The attended features regress $a_i$ and the gate $g_i$. The fused $d_i$ enter the differentiable ARAP solver as soft correspondence targets, and the whole system is trained end to end through the solver.
Training signal. Dense ground truth is sparse on real data, so the tangential corrections are learned mainly from a self-supervised cycle loss over short windows. For a node at frame $t$, let $x_i^{t\to t+2}$ be its position tracked directly to frame $t+2$, and $x_i^{t\to t+1\to t+2}$ its position composed through frame $t+1$. The loss is
$$\mathcal{L}_{\text{cyc}} = \frac{1}{|\mathcal{V}|}\sum_{i,t} \min\!\left(\big\|P_i\,(x_i^{t\to t+2}-x_i^{t\to t+1\to t+2})\big\|^2,\ \tau^2\right).$$
It penalizes only the tangential inconsistency, since normal-direction errors are already visible to geometry. The truncation $\tau$ stops tears, fast motion and occlusion from dominating. We extend windows to hops of 4 and 8 frames, because sliding is small per step and only becomes measurable across hops. We set $\tau$ from a percentile of the per-node residuals on a held-out split rather than tuning it by hand, and we report sensitivity to that percentile, since the file identifies threshold tuning as a known weakness of truncated losses.
This loss is not a guarantee against drift. A consistent slide in the same direction at every hop passes the cycle test. That is why the color branch, rather than the loss, must supply the anchoring information. It is also why the evaluation separately measures drift against ground truth.
Analysis. We will measure how the gate $g_i$ and the projector $P_i$ distribute over curvature, texture strength and motion speed. This tests whether the network learned "color where geometry is blind" or something else.
Baselines
The naive reference point is geometry-only tracking. We run Neural Non-Rigid Tracking's solver with the color input removed, same graph, same training data and schedule. This quantifies how much sliding exists to be fixed on each benchmark. If its tangential error is not substantially larger than its normal error on smooth regions, the premise fails on that data.
The strongest existing method is Neural Non-Rigid Tracking as published. It reports a 3D deformation error of 28.72 mm on the DeepDeform benchmark, against 31.52 mm for the previous best. DeepDeform reports 3.28 cm for its own pipeline. For fairness we retrain Neural Non-Rigid Tracking on identical splits with the same augmentation, training steps, and hyperparameter search budget as our model. We report both our reproduction and the published number. Beating it means a lower deformation error than our reproduction, with non-overlapping seed intervals, and a clearly lower tangential drift on the long-horizon evaluation below.
Two fusion controls isolate what the projector contributes.
- Channel concatenation. The same backbone and solver with color and depth concatenated at the input. This is the DeepDeform-style fusion, retrained under the same budget.
- Unrestricted gated attention. Our exact model with $P_i$ replaced by the identity. The gate stays learnable in the style of gated cross-attention. This is the most important ablation: it has the same parameters, compute and loss, and lacks only the geometric restriction.
A further ablation removes $\mathcal{L}_{\text{cyc}}$, and another uses it untruncated. Each separates the temporal loss's contribution from the fusion's.
Why not just…?
Why not just concatenate RGB and depth, as DeepDeform does? Concatenation lets the network use color, but gives it no reason to use color specifically in tangential directions. Unchecked, color can also override correct geometry under lighting changes. The file cites evidence that concatenation underperforms attention-based fusion in frame-event tracking and histology (How GNNs Facilitate CNNs in Mining Geometric Information from Large-Scale Medical Images). That evidence comes from other tasks, though, and is not decisive here. The concatenation baseline, measured specifically on tangential error in low-curvature regions, settles it.
Why not just add a photometric term to the ARAP energy? This is the most serious threat, because a color-constancy term is classical and cheap. Its weakness is also predictable. It is local and gradient-based, so on the untextured skin and fabric that cause sliding it has no gradient. Under shading changes it is biased in every direction, including the normal direction where geometry was already right. We will implement it with its weight tuned on validation under the same budget as our model. If it matches our tangential-drift numbers, the attention component is unnecessary. We would then reframe the contribution as the null-space projection applied to a photometric term.
Why not just smooth the tracked trajectories afterward (for example with Savitzky-Golay filtering)? Sliding is a slow, low-frequency bias, not jitter. A temporal filter preserves low-frequency content by design, so it cannot remove this error. We include the filter on top of the geometry-only tracker and expect it to reduce frame-to-frame jitter while leaving drift at 50 and 200 frames unchanged. If drift does drop, sliding on that data is noise-like and our motivation is weaker.
Why not just let a learned gate decide, without the curvature projector? A gate could, in principle, discover null spaces from data. But the gate sees only one scalar per node, not a direction, so it can only choose whether to trust color, not along which axes. The unrestricted-gate ablation tests this directly. If it matches the full model, the projector is redundant and we will say so.
Proposed Experiments
DeepDeform. On DeepDeform we report 3D deformation error and 2D matching error under its standard protocol, comparing against the reproduced and published numbers above. This establishes that restricting fusion does not cost accuracy on the field's reference benchmark.
D-FAUST. The decisive experiment uses Dynamic FAUST. Its painted-texture registrations give dense, surface-consistent ground truth, with 82% of scan points aligned within 1 to 2 mm. We render depth and color from its sequences in two conditions. In the first, the original painted texture is visible. In the second, the paint is suppressed by low-pass filtering the color toward natural skin appearance.
Tracking starts at a reference frame. We measure vertex error at horizons of 1, 10, 50 and 200 frames, split into the tangential and normal components with respect to the ground-truth surface, and binned by local curvature. No learned-tracker result on this rendered protocol is reported on Lacuna, so all numbers will be produced here under one protocol.
Our hypothesis predicts a specific pattern. The advantage over the unrestricted-gate and concatenation controls should appear mainly in the tangential component, grow with horizon, and concentrate in low-curvature bins. It should shrink but not vanish in the suppressed-texture condition. A uniform improvement across components, or one confined to the 1-frame horizon, would contradict the mechanism even if the headline error improves.
CAPE. On CAPE clothed-human scans, used zero-shot by GraphSCNet without a reported number on Lacuna, we test transfer to loose, shifting fabric. Occasional topology changes here also exercise the truncation.
Efficiency. We report per-frame runtime against Neural Non-Rigid Tracking's 27 ms per keyframe, since a local-window attention stage must not erase its speed advantage.
References
- Dynamic FAUST: Registering Human Bodies in Motion
- Neural Non-Rigid Tracking
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- DeepDeform: Learning Non-Rigid RGB-D Reconstruction With Semi-Supervised Data
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
- MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
- Deep Graph-based Spatial Consistency for Robust Non-rigid Point Cloud Registration
- Lepard: Learning partial point cloud matching in rigid and deformable scenes
- Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery
- 3D Implicit Transporter for Temporally Consistent Keypoint Discovery
- Object Tracking by Jointly Exploiting Frame and Event Domain
- How GNNs Facilitate CNNs in Mining Geometric Information from Large-Scale Medical Images