Censored Endpoints: 3D Line Mapping When Things Move

Man-made scenes such as corridors, offices and streets are full of straight edges, and those edges are often the only reliable geometry where walls are blank. Systems that recover 3D line segments from many photos have one weakness. They treat every place a detected 2D segment stops as a place where the physical edge stops. When a person walks past a door frame, the detector cuts the frame's edge at the person's silhouette. The mapping system then either shortens the 3D line, splits it into pieces, or throws it away because it touched a "dynamic" mask. We propose to label every 2D endpoint as either a true end, meaning the edge physically stops there, or an occlusion end, meaning the edge merely disappeared behind something. Occlusion ends are then treated the way survival analysis treats patients who left a study early: as censored observations that say "the line extends at least this far," not "the line ends here."

The two-sentence version is this. Line fragmentation in dynamic scenes is mostly a problem of mislabeled endpoints, not of bad detectors. If each endpoint carries a physical-versus-occlusion label, estimated from image appearance, depth and a motion signal, then 3D line extent can be estimated with a censored likelihood. A learned prior over where man-made lines actually terminate fills in the stretches that no frame ever saw.

Core Claim

Current 3D line mappers fall into two camps. In the first, the static scene assumption is baked into how endpoints are interpreted. In the second, dynamic-scene methods drop line geometry altogether. A single change fixes both: treat each endpoint as an observation with a type. For physical ends, the type is "exact." For occlusion ends, the type is "censored," and the true extent lies beyond the observed point in the direction the edge was heading. This one change should produce three effects. It should raise line recall in scenes with moving occluders. It should prevent false merges across genuine gaps such as doorways, while allowing merges across occluder-induced gaps. It should also give structural completion a principled place to act, because only censored ends are allowed to be extended by the prior.

Where This Sits

The fragmentation problem is well documented. LSD groups pixels by gradient direction and routinely truncates edges. Work such as Line-sweep and Robust Incremental Structure-from-Motion with Hybrid Features notes that endpoints are not repeatable landmarks across views. Two mappers supply the machinery we reuse:

  • LIMAP triangulates and tracks segments across views.
  • CLMAP adds collinearity-constrained Union-Find tracking and joint optimization with planes and vanishing points. It reports 79.7% Length Recall on Hypersim against LIMAP's 32.8%.

Both assume a static world. Neither distinguishes why a segment stopped. CLMAP's collinearity merge will happily join fragments across a real doorway gap, or refuse to join them across a person-shaped gap, depending only on geometric thresholds.

NEAT takes the matching-free route. It distills wireframes from a neural attraction field inside a signed-distance volume renderer and reports junction accuracy of 0.7718 on DTU. Its volume rendering, however, is invalidated by moving content.

On the dynamic side, three systems either handle lines crudely or not at all:

  • IPL-SLAM uses lines only in a hand-set angular-deviation test for dynamic features and outputs a point map.
  • CAD-SLAM detects motion by comparing a render of the historical static map against the current frame, with no structural primitives.
  • RobustSplat simply discards transients.

We borrow CAD-SLAM's render-residual idea as a motion cue that needs no object detector. This matters because Split4D and CAD-SLAM both argue that external segmentation errors propagate into 3D.

For the structural prior we draw on UniCo. It predicts primitives in one learned pass rather than fitting them afterward with RANSAC, and it shows this degrades gracefully under 75% missing data. We apply the same idea in one dimension: a learned distribution over where a line terminates, given the surrounding structure.

For the endpoint classifier we follow SGFormer's finding that geometry-as-query cross-attention beats naive concatenation under misalignment. We also follow Countering Multi-modal Representation Collapse's warning that fused models lean on appearance and ignore geometry.

Research Program

Endpoint labeling. For each 2D endpoint we gather three inputs:

  • a small RGB patch around it;
  • a depth profile across the endpoint, from the RGB-D sensor where available and otherwise from monocular depth;
  • a motion residual, computed CAD-SLAM style from the current partial static map.

A small transformer uses a query token at the endpoint to attend over these three modalities. It outputs the probability that the end is physical. The depth branch encodes the key physical signature: at an occlusion end, the side beyond the endpoint is nearer to the camera than the edge itself, and often moving. Training labels come for free on synthetic data. Using ground-truth depth and instance ordering, we check whether the rendered edge continues behind a nearer surface.

Censored extent estimation. After CLMAP-style tracking groups fragments into a 3D line hypothesis with direction $d$, each observation $i$ projects its endpoint to a position $t_i$ along $d$. Let $p_i$ be the classifier's probability that endpoint $i$ is physical, $b$ the unknown true end, $f_\sigma$ a noise density of width $\sigma$, $S_\sigma$ its survival function (the probability that the true value exceeds a given point), and $\pi(b)$ the structural prior. We estimate $b$ by maximizing

$$ \log \pi(b) + \sum_i \log\big[\, p_i\, f_\sigma(t_i - b) + (1-p_i)\, S_\sigma(t_i - b) \,\big], $$

and do the same for the other end. Physical ends pull $b$ toward $t_i$. Censored ends only push $b$ outward. If all observations of an end are censored, the prior $\pi$ alone decides where the line stops.

Structural prior. The prior $\pi(b)$ is learned from static line maps. It is a density over candidate stopping points along the line: intersections with other 3D lines, plane boundaries, and vanishing-direction-consistent junctions. Its parameters are predicted from local structure, in the spirit of UniCo's proxies but for one-dimensional extent.

Merge rule. The same labels drive merging. Two collinear fragments are joined only if the facing endpoints of their gap are predominantly censored. We plug this into CLMAP's tracker in place of its purely geometric merge threshold.

Baselines

Do-nothing reference. We run LIMAP on raw frames with no dynamic handling. This measures how badly moving content corrupts an unmodified line mapper.

Mask-and-triangulate. We run an instance segmenter, drop every segment touching a dynamic mask with IPL-SLAM-style mask expansion, and feed the rest to CLMAP. This is the pipeline a practitioner would build today.

Strongest existing method. CLMAP with the same motion-residual masking and a max-extent union rule, where each 3D line takes the farthest endpoints any view observed. This is the strongest competitor because it already merges collinear fragments and holds the best reported static recall: Length Recall 79.7% with DeepLSD on Hypersim, per CLMAP. Union-of-extents also silently recovers any part of the line that some frame saw unoccluded.

Matching-free baseline. NEAT trained on frames with RobustSplat-style transient masks applied to its rendering loss. This tests whether the matching-free route escapes the problem without endpoint reasoning.

Fairness. All methods use the same DeepLSD detections, the same camera poses (ground truth for mapping experiments), the same motion masks where masks are used, and the same per-method threshold search on a held-out validation split.

What counts as beating CLMAP-union. Higher Length Recall at equal or better precision on the dynamic benchmarks. This win must be concentrated on lines that were ever occluded, with no loss on static-only scenes.

Ablations. These isolate the key ingredient:

  • censoring with a hand-coded depth-jump rule instead of the learned classifier;
  • learned labels without the structural prior;
  • the prior applied to all ends without censoring labels;
  • the classifier without its depth branch or without its motion branch.

Why not just…?

Why not just take the union of observed extents across frames? This is the most serious threat. When an occluder moves, some frame usually sees each part of the edge, so union already recovers extent for free. It fails in three situations:

  • The occluder lingers, or the camera views the edge only while it is occluded. Union then has no complete observation, and only censoring plus the prior can extend the line.
  • Union also extends lines past spurious long detections, such as edges of the moving object itself that happen to be collinear with background edges. This costs precision, which union has no way to reject.
  • Union says nothing about merging across gaps.

We will report results stratified by the fraction of frames in which each ground-truth line is fully visible. If our method's gain vanishes except when full visibility is below about 20%, we will say so and position the method for those regimes: surveillance-like cameras and brief handheld captures.

Why not just mask dynamic objects and discard touching segments? Masking creates pseudo-endpoints at the mask boundary and removes whole segments, which lowers recall. CLMAP's own comparison shows how quickly recall collapses without careful tracking (LIMAP 32.8% versus CLMAP 79.7% on static Hypersim). Masks from category detectors also miss open-world movers, as Split4D and CAD-SLAM argue. The mask-and-triangulate baseline measures this directly.

Why not just use a depth-discontinuity rule instead of a learned multimodal classifier? It may work nearly as well when good RGB-D depth is available, and our ablation will reveal this. We expect the rule to fail at moving shadows, which change appearance but not depth, and in monocular settings where depth jumps are blurred. If the rule matches the classifier, the contribution reduces to censored estimation plus the prior. That result would still be worth reporting, and we would adopt the simpler rule.

Why not just rely on photometric dynamic Gaussian splatting to inpaint the background? EnerGS shows that photometric optimization alone leaves floaters (a leak ratio of 12.5 for vanilla 3DGS on KITTI). Photometric inpainting behind occluders on blank walls has no gradient to place a sharp edge. We will extract edges from CAD-SLAM's static map on Bonn and compare line precision against ours.

Proposed Experiments

Primary benchmark. We build a dynamic version of Hypersim: animated human and object layers are depth-composited into the provided RGB and depth sequences. This preserves Hypersim's ground-truth 3D lines and yields exact endpoint labels. We report Length Recall and precision at several distance thresholds, overall and stratified by occlusion fraction and dwell time. We also evaluate on unmodified Hypersim and check that we do not fall below CLMAP's 79.7%. No reported result found on Lacuna for any line mapper on dynamic data, so all dynamic numbers will be produced by us under the shared protocol above.

Real data: Bonn. We score precision of reconstructed static lines by sampling points along each line and measuring their distance to Bonn's static ground-truth surface. For reference, CAD-SLAM reports 6.30 cm average surface reconstruction accuracy on this dataset for its dense map; this is a dense-map number, not a line metric.

Secondary test: TUM fr3/walking. We add our lines as residuals in pose tracking to test whether complete static lines reduce drift. The reference number is IPL-SLAM's absolute trajectory error of 0.0129 m RMSE on fr3/w/xyz.

Wireframe accuracy. We check junction accuracy on DTU, where NEAT reports ACC-J 0.7718, to confirm that the structural prior does not distort static geometry.

Decisive checks. The proposal succeeds if three conditions hold on dynamic Hypersim:

  • it beats CLMAP-union on ever-occluded lines at matched precision;
  • the no-censoring ablation loses most of that gain;
  • the merge rule reduces false merges across true gaps relative to CLMAP's geometric threshold.

It fails, and we would say so, if union matches it everywhere except scenes with near-zero full visibility.