Learning How Likely Things Are to Move: Open-Vocabulary Mobility Priors for Dynamic SLAM

A robot that maps its surroundings while tracking its own position, a task called SLAM (simultaneous localization and mapping), assumes the world is still. When people walk past, carts roll by, or a chair gets pushed aside, that assumption breaks, and the robot's position estimate drifts. The usual fix is to run an image segmenter, look up whether each detected object belongs to a hand-written list of "dynamic" classes such as person or car, and throw away all measurements on those objects. This has two failures. The segmenter only knows a fixed set of categories, so a dog or a forklift is never flagged. The list is also binary, so a seated person or a parked car is discarded even though it is a perfectly good stationary landmark. Newer systems drop the class list and instead down-weight regions whose visual features change across views. This handles unknown objects, but it ignores something language models know well: a "cat" is likely to move and a "load-bearing column" is not.

We propose replacing the binary class list with a learned, continuous mobility prior: a small model that reads an open-vocabulary embedding of each image segment and outputs a probability that the segment will move. The prior is trained on motion that was actually measured in datasets with ground-truth camera poses, not on anyone's class list. It sets the starting belief for each segment, and per-segment geometric evidence accumulated over the tracking window then updates that belief. The weight each segment receives in pose optimization is its posterior probability of being static. The prior lets the system distrust a never-seen animal from its first frame. The evidence lets it keep a parked car that a class list would discard and reject a chair as soon as someone moves it.

Core Claim

Whether an object counts as dynamic is a label that prior systems define inconsistently. DS-SLAM treats people as dynamic. DynaSLAM adds "movable" objects. Agricultural and reef systems (MOLO-SLAM, DeepReefMap) need their own lists again. This is the same taxonomic friction that multi-dataset segmentation faces, applied to a single property. We claim that supervising with measured motion rather than class names, and expressing the prior over a continuous vision-language embedding, gives two things. First, it transfers to object categories absent from training. Second, combined with a sequential geometric update, it beats both closed-set masking and prior-free embedding-discrepancy weighting on scenes with unfamiliar movers and paused movers. It should not lose accuracy on the standard walking-person sequences where closed-set masking already works.

Research Program

Segments and embeddings. Following the "where/what" split of MaskFormer, we generate segment proposals without class labels and attach one embedding per segment, using a side-adapter design in the spirit of SAN (8.4M trainable parameters, 15.2 FPS reported) to keep cost bounded. Working with whole segments rather than dense per-pixel embeddings matters here. RADIO-ViPE couples dense embeddings directly into its optimization and reports trouble segmenting walls and floors. Our prior does not need a correct name for a wall. It only needs to give it a low mobility score. Structural classes are exactly the anchors pose estimation needs, so this distinction is useful.

Training the prior without manual labels. We use sequences that come with ground-truth camera trajectories, namely TUM RGB-D, ARIA Digital Twin and KITTI. For each segment over a short window, we compute the reprojection residual left after subtracting the true camera motion. Segments whose residual exceeds a noise threshold are labeled as having moved in that window. The prior network $f$ maps a segment embedding $e$ to $\pi = f(e)$, the probability of motion in a window. It is trained with a calibrated binary loss on these labels.

We handle the label-inconsistency problem in two ways. First, motion is measured physically, so no dataset's dynamic-class definition enters the targets. Second, when we add segmentation datasets with conflicting category granularity to improve the embeddings, we use text-embedding targets as in LMSeg rather than a hand-merged taxonomy like MSeg. LMSeg reports an average mIoU of 54.75 against 54.53 for a manual taxonomy. LMSeg was never tested on unseen concepts, and testing transfer to unseen concepts is exactly what our held-out-category experiments do.

Sequential update. We track each segment $i$ with a log-odds belief $\ell_{i,t}$ that it is static at time $t$. The belief is initialized from the prior and updated with each new residual $r_{i,t}$, in the same form DS-SLAM uses for map voxels:

$$ \ell_{i,0} = \log\frac{1-\pi_i}{\pi_i}, \qquad \ell_{i,t} = \ell_{i,t-1} + \log\frac{p(r_{i,t}\mid \text{static})}{p(r_{i,t}\mid \text{moving})}. $$

The two residual likelihoods are fit on the same ground-truth-pose data. Feature points on segment $i$ enter bundle adjustment, the joint optimization of camera poses and 3D points, with weight $\sigma(\ell_{i,t})$, the posterior probability that the segment is static. We clamp the log-odds so that a long pause cannot make a high-prior object fully trusted. This prevents a paused pedestrian from being trusted one frame before it walks off.

A secondary modality as a stress test. A second sensor stream is a robustness test, not the core of the method. We add a lightweight thermal branch that injects thermal features into the frozen RGB encoder, following the prompt-injection design of OpenRSS. We then ask whether the prior stays calibrated when RGB degrades. Body and engine heat are plausible mobility cues, and darkness is where RGB-only open-vocabulary SLAM such as KM-ViPE breaks down. No dynamic RGB-thermal SLAM benchmark appears in our sources. This part is therefore evaluated per frame, by scoring mobility predictions against motion on MF, MCubeS and Freiburg Thermal, and is labeled exploratory.

Baselines

The naive reference point is ORB-SLAM2 with only its standard geometric outlier rejection. DS-SLAM reports 0.7521 m ATE RMSE (absolute trajectory error, root mean square) for it on TUM fr3/walking_xyz. Towards Real-time Semantic RGB-D SLAM reports 0.4662 m. We rerun it on all our sequences.

The strongest closed-set competitor is the system from Towards Real-time Semantic RGB-D SLAM. It pairs SegNet masking with depth clustering to catch unknown movers. It reports 0.0194 m ATE on fr3/walking_xyz at about 75 ms per frame on a Jetson AGX Xavier, which is the best number in our sources for that sequence. DS-SLAM's 0.0247 m is a close second. On standard walking sequences we aim only to match these, within run-to-run variance over five seeds. We count it as beating them if we reduce ATE on sequences containing novel or paused movers, where their fixed class list cannot help.

The strongest open-vocabulary competitor is the adaptive robust kernel of KM-ViPE and RADIO-ViPE. It down-weights regions whose embeddings change across views and uses no mobility prior. Neither paper's summary gives a numeric ATE, so we found no reported result on Lacuna for KM-ViPE or RADIO-ViPE on TUM. To compare fairly, we reimplement their discrepancy-driven weighting inside our own back end, using the same segments, the same embedding model and the same tracking window. Only the weighting rule then differs, and we also run their released systems where available.

Two baselines target the proposal directly. The first is a word-list baseline. Our open-vocabulary segmenter is prompted with a generous list of mobile nouns, such as person, animal, vehicle, cart and robot, and matching segments are hard-masked. This list gets the same prompt-engineering budget as our prior's training. The second is a set of ablations: prior only, with no sequential update; evidence only, with a flat prior, which is essentially the geometric half of our system; and the full system with the prior trained on class-list labels instead of measured motion. That last ablation isolates whether motion supervision is the key ingredient. For the thermal stress test we compare against channel stacking of RGB and thermal into one backbone, at matched trainable parameters.

Why not just…?

Why not just prompt an open-vocabulary segmenter with a list of moving things and mask them? This is the most serious threat, because it already fixes the "unknown class" problem for any noun someone thinks to write down. It cannot express the other failure: a hard mask discards a seated person or parked car, and it never rejects a chair that gets pushed. The word-list baseline and the evaluation splits for paused and moved objects test this directly. If the word list matches our system on those splits, the learned prior adds nothing beyond the sequential update. We would then report the contribution as "open-vocabulary masks plus log-odds evidence" and drop the learned-prior claim.

Why not just rely on cross-view embedding discrepancy, as KM-ViPE and RADIO-ViPE do? Discrepancy only appears after an object has moved, and it can stay small for slow or rigid movers whose appearance hardly changes. A prior acts from the first frame. Our evidence-only ablation, together with a newly-entering-object metric, measures ATE in the first frames after a mover appears and shows whether that early advantage is real. If it is not, the prior is not worth its cost.

Why not just use geometry with a robust loss and skip semantics? Geometry-only tracking is the 0.75 m baseline above. NID-SLAM similarly reports 0.629 m for NICE-SLAM against 0.071 m with semantic masks on fr3/w/half. Geometry fails when movers fill the view, and it cannot tell a paused vehicle from a wall. The evidence-only ablation reproduces this regime inside our own pipeline.

Why not just stack thermal as an extra input channel? Zhang et al. and OpenRSS report that parallel branches beat concatenation, and OpenRSS reports gains of 12.1, 18.4 and 21.4 percent mIoU on MF, MCubeS and Freiburg Thermal. Those are segmentation results, not mobility calibration. We run the channel-stacking comparison rather than assume the answer.

Proposed Experiments

We evaluate on the TUM RGB-D dynamic sequences, including the low-motion sequences in the same benchmark where people sit, using ATE RMSE. We also use the ARIA Digital Twin dynamic sequences and KITTI, scoring ATE plus mobility-mask quality: the AUROC (area under the ROC curve) of predicted motion against residual-derived motion labels, and expected calibration error. Replica tests that static-scene accuracy and depth L1 do not regress. We report runtime per frame on desktop hardware and on a Jetson-class device, with 8–10 FPS from RADIO-ViPE as the open-vocabulary reference and 75 ms from Ji et al. as the embedded reference.

The decisive check is leave-category-out training. We remove every segment whose nearest text label is "person", or a related term, from the prior's training data. We then evaluate on TUM walking sequences and on ARIA sequences with movers whose categories were never in training. If the prior still assigns these segments high mobility and the full system's ATE stays near the closed-set numbers, language-space transfer is real. If ATE collapses toward the evidence-only ablation, the prior is only memorizing categories it has seen. That would also confirm, for mobility, the kind of training-class bias OVO-SLAM reported for its embedding-merging network, and the claim fails.

The second check uses the paused-mover and moved-furniture splits. On these, the full system must beat both the word-list mask and DS-SLAM-style masking. Otherwise the continuous prior plus sequential update is not justified.