World-Anchored Region Graphs for RGB-D Video Object Segmentation

Depth cameras make one part of foreground segmentation easy. A shadow, a dark room, or a target the same color as its background changes how a pixel looks, but it does not change how far away the surface is. Video segmentation models that use depth still tend to drift over long sequences, though, and the fastest-growing fix is expensive. On the mobile-LiDAR benchmark ARKitTrack, the strongest reported RGB-D system calls the Segment Anything Model (a large, general-purpose image segmenter) to repair its masks and runs at about 1.5 frames per second. We propose to put depth in a different place. Depth should decide which parts of the scene are allowed to exchange information, rather than serving only as an extra input feature.

Concretely, we cut each frame into small regions that are consistent in depth. We lift each region into 3D world coordinates using the camera poses that phones and tablets already record. Each region becomes a node in a graph, and a node's foreground/background state is carried forward through time. Edges link two regions only if they are physically close in 3D within a frame, or if they occupy the same place in the world across frames. The memory each node carries is updated by a stable recurrent rule. That rule forgets quickly where the depth reading is unreliable and holds information where the depth is clean. The bet is that this sparse, physically grounded graph resists drift as well as heavy foundation-model correction does, at a small fraction of the cost.

Core Claim

Our main claim concerns graph structure, not the mere presence of depth. In semi-supervised RGB-D video object segmentation, the setting where the target's mask is given in the first frame and must be tracked thereafter, sparse message passing over a graph whose edges come from metric 3D proximity in a world-fixed frame should reduce long-horizon drift relative to two alternatives. The first alternative is dense attention over memory features, even when that attention is softly biased by depth. The second is graphs built from 2D image overlap.

We pair this with a second claim. Noisy depth should be handled by a per-node forgetting rate predicted from local depth reliability, rather than by a single frame-level accept/reject decision. If the first claim fails, meaning a depth-biased dense attention baseline matches the sparse graph, then hard topology is not what matters, and we will say so.

Relation to Prior Work

Two papers motivate the proposal by doing half of it each. Spatiotemporal Graph Neural Network based Mask Reconstruction for Video Object Segmentation (STG-Net) treats Mask R-CNN proposals as nodes of a spatial graph and links reconstructed masks to past masks through a temporal graph. It reaches a global mean score of 63.1 on DAVIS 2017 test-dev at about 6 FPS. Its edges come from 2D appearance similarity and 2D box overlap. As a result, it cannot separate two objects that overlap in the image but sit at different depths, and it has nothing to work with when the 2D detector fails in the dark. Project-and-fuse: Improving RGB-D semantic segmentation via graph convolution networks builds a region graph over single RGB-D images, converting depth to surface normals or HHA maps and adding Euclidean region distances to edge weights. It reports 50.6% (normals) and 50.8% (HHA) mIoU on NYUDv2. That paper is static, and its authors report that adding nodes is costly. We take from it the region-graph idea and the normal-map input, and we take the temporal graph from STG-Net. What we add is a topology defined in world coordinates, which neither paper has.

Three RGB-D video systems constrain the design and set the bar. RGB-D Video Object Segmentation via Enhanced Multi-store Feature Memory extends the XMem memory network with modality weighting, a depth-entropy check that rejects noisy depth frames, and SAM-based mask repair. It reports $J\&F$ = 0.698 on ARKitTrack at about 1.5 FPS. The ARKitTrack baseline lifts depth to a bird's-eye-view grid and reports $J\&F$ = 0.662 on ARKitTrack-VOS. It shows that keeping geometry in 3D helps, but it updates temporally only through a template. DVSOD shows that depth plus temporal memory lifts F-measure from 0.548 to 0.610 on DViSal. Its authors note that pixel-level memory does not enforce temporal coherence.

The classical alternative is Temporally Consistent Motion Segmentation From RGB-D Video. It gets global consistency by energy minimization, but it assumes rigid motion and needs up to 24 hours for 220 frames. We want its consistency without either restriction.

For the recurrent update we borrow from Adaptive Memory Retention in Dynamic Graphs (LAMP). LAMP treats graph snapshots as impulses on a continuous state whose dynamics are skew-symmetric, so signals neither blow up nor fade, plus a learned damping term. It reports test MSE 0.223 against DCRNN's 0.675 on a 600-step memory task. The adaptive edges of Adaptive Spectral Graph Convolutional Networks for Skeleton-Based Action Recognition (2s-AGCN) supply a way to learn data-dependent connections for non-rigid targets. We allow such connections only among nodes that are already near each other in 3D.

Research Program

Graph construction. Each frame is over-segmented into a few hundred regions. Each region is a patch whose depth and surface normals are smooth inside it, with RGB edges used only where depth is missing. Each region is back-projected using the camera intrinsics and the 6-DoF pose that ARKitTrack provides per frame, giving a 3D centroid, an extent, and a mean normal in world coordinates.

Spatial edges connect regions that share an image boundary, but only when the depth jump across that boundary is below a learned threshold relative to sensor noise. Additional edges connect regions within a metric radius. Temporal edges connect a region at time $t$ to regions at $t-1$ whose world-frame extents overlap after the target's estimated motion is compensated. Because static background regions stay put in world coordinates, they link cleanly across time. This matters because, in practice, most drift comes from the background leaking into the target.

Node features. These are pooled from a standard RGB encoder and a normal-map encoder. Each node also carries a reliability score $r_i$, built from the fraction of valid depth pixels in the region, the local depth variance, and the distance to the nearest depth hole.

Node update. Each node carries a hidden state $h_i$, updated between observations by

$$h_i \leftarrow h_i + \epsilon\Big[(W - W^\top)\,h_i - \gamma(r_i)\,h_i + \sum_{j \in \mathcal{N}(i)} \alpha_{ij}\,\phi(h_j)\Big],$$

where $W$ is a learned matrix whose skew-symmetric part $W - W^\top$ preserves signal norm, $\gamma(r_i)\ge 0$ is a damping rate predicted from the node's depth reliability, $\mathcal{N}(i)$ are its 3D spatial and temporal neighbors, $\alpha_{ij}$ are attention weights restricted to those neighbors, $\phi$ is a small network, and $\epsilon$ is a step size. A new frame acts as an impulse that writes the node's fresh features into $h_i$, scaled by $r_i$.

The intent is that a region inside a depth hole or along a reflective surface forgets its unreliable evidence and leans on its neighbors. A region with clean depth keeps its identity across occlusions.

Decoding. Per-node foreground probabilities are painted back onto pixels. A light RGB decoder then refines the boundaries, since depth is least reliable exactly at object edges.

Training. The model is trained end to end on the ARKitTrack training split, using the first-frame mask as the only target supervision at test time and no online fine-tuning.

Baselines

The naive reference point is first-frame mask propagation by depth gating. We take the target's depth range from the first mask, keep pixels in that range near the previous mask, and update frame by frame. This tells us how much of ARKitTrack is solved by depth thresholding alone, the classical recipe behind the grapevine and diminished-reality results the file cites. It will be run with tuned thresholds on the validation split.

The strongest existing method is the multi-store memory system of RGB-D Video Object Segmentation via Enhanced Multi-store Feature Memory. It reports $J\&F$ = 0.698, $J_M$ = 0.673, and $F_M$ = 0.723 on ARKitTrack at about 1.5 FPS on an RTX 3090. We will report its published numbers. We will also rerun its released configuration, if one is available, on the same split and hardware to measure speed under identical conditions. Beating it outright means exceeding 0.698 $J\&F$. We will also count it as a practical win if we come within one point of $J\&F$ while running at least ten times faster on the same GPU, and outperform it on the occlusion and depth-clutter attribute subsets.

The ARKitTrack bird's-eye-view baseline at $J\&F$ = 0.662 is the strongest method that keeps 3D geometry without a foundation model. Beating it is the minimum bar.

Two ablations isolate the key ingredient, and both share our encoders, training data, and tuning budget. The first is an STG-Net-style variant. It has the same nodes and update rule, but its edges come from 2D adjacency and 2D overlap, with depth used only as a node feature. If it matches the full model, the 3D topology is not doing the work.

The second is the more dangerous one: a depth-biased dense memory network. This is XMem's memory reading with an additive attention bias proportional to negative 3D distance between memory and query locations, without SAM. It is the soft, dense version of our sparse graph, so it directly tests whether hard sparsity and world anchoring matter.

Two further ablations replace $\gamma(r_i)$ with a single learned constant, and replace world-frame temporal edges with camera-frame edges.

Why not just…?

Why not just add depth-aware bias to the attention of an existing memory network like XMem? This is the most serious threat. Dense attention with a 3D-distance bias can in principle learn everything our sparse graph encodes. The file contains no reported result for this exact variant, so we build it as a baseline with matched compute. Our reasons to expect a gap are drift and noise. Dense attention lets every noisy memory entry contribute a little, and Adaptive Memory Retention in Dynamic Graphs shows that unconstrained propagation accumulates noise over long horizons. If the biased dense model matches ours on late-sequence frames and on the depth-corruption test, we will reframe the contribution around the reliability-gated forgetting rule, which can be bolted onto either model.

Why not just gate an RGB-only video segmenter's output by the target's depth range? This works when the target floats in free space. It should fail when the target touches a surface at the same depth, such as a hand on a table or a cup against a wall, and when depth has holes. The first case is exactly where adjacency-plus-depth-jump edges and normal-based splitting help. The naive baseline above measures this directly, and the depth-clutter subset of ARKitTrack shows where it breaks. If gating alone comes within a few points of the full model overall, the graph is not earning its cost on this benchmark.

Why not just use the SAM-repaired memory system and accept the speed? For offline annotation that is reasonable. For phones and robots, 1.5 FPS is not (RGB-D Video Object Segmentation via Enhanced Multi-store Feature Memory). The sparse graph over a few hundred nodes costs far less than a ViT-H forward pass. The speed-matched comparison above, plus a variant of our model that may also call SAM on low-confidence frames, shows whether the graph and SAM are complementary or redundant.

Why not just run classical rigid energy minimization over the sequence? It needs up to 24 hours per 220 frames and assumes rigid parts (Temporally Consistent Motion Segmentation From RGB-D Video). ARKitTrack's 287 dynamic scenes include people and other deformable targets. We do not rerun it at full scale. Its runtime alone excludes it from streaming use, and we will say that rather than claim an accuracy comparison we have not made.

Proposed Experiments

The primary benchmark is ARKitTrack-VOS, with region similarity $J$, contour accuracy $F$, their mean $J\&F$, and frames per second on an RTX 3090. Results are reported overall and on the dataset's Depth Clutter, Extreme Illumination, and Occlusion attributes.

The decisive check for the drift claim plots $J\&F$ against frame index and against frames since the most recent full occlusion. The prediction is that the gap between our model and both the 2D-topology and dense-attention ablations grows with time. If it is flat, the drift claim is wrong.

The decisive check for the reliability claim is a controlled corruption test. We delete depth along object boundaries, punch random holes, and add multi-path-like offsets at increasing rates on held-out sequences. The prediction is that the learned $\gamma(r_i)$ degrades more gracefully than both the constant-damping ablation and the frame-level entropy rejection used by Xu et al.

Because the method depends on camera poses, we inject pose noise at levels typical of mobile tracking failure, and we compare against camera-frame edges. This tells us whether the world anchoring is robust or brittle.

As a secondary test of generality without poses, we run a pose-free variant, with temporal edges in the camera frame, on DViSal for RGB-D video salient object detection. The comparison point there is the reported F-measure of 0.610 from DVSOD, along with mean absolute error and a frame-to-frame mask stability score that measures the temporal coherence its authors flag as missing.

To check that the region graph is a sound per-frame module, we evaluate a single-frame version on NYUDv2-40 against the 50.6% mIoU that Project-and-fuse reports with normal-map input. This is a sanity check, not a headline result.