Seeing Nothing Is a Measurement: Occlusion-Truncated Particle Proposals for Control-Rate Object Tracking

When a robot loses sight of an object because it passes behind something, most trackers treat that moment as "no data." They coast the last estimate forward with a constant-velocity guess and wait for the detector to fire again. Yet a depth camera never stops measuring. Even while the object is hidden, every pixel reports how far away the nearest visible surface is, which says exactly where the object cannot be: anywhere in front of that surface along the camera's line of sight. This note proposes a particle filter that uses that negative information to decide where to place its guesses during occlusion, rather than only to score them after the fact. A particle filter tracks an object with a cloud of weighted guesses called particles. The tracker must also run inside the 1–10 ms cycle of a robot controller, and it must absorb pose detections that arrive 100–300 ms late from heavy neural networks.

The core mechanism is simple. We re-express each particle's predicted position in camera coordinates: a pixel plus a depth along the viewing ray. When the object is undetected, we sample the depth from the motion prediction cut off so that only depths behind the observed surface are allowed. This is a truncated Gaussian. The particle's weight then becomes the probability the prediction assigned to "behind the surface," which has a closed form. In a precise sense this is the best possible way to place particles for this measurement. It costs one depth-image lookup and one normal-distribution evaluation per particle, and no guess is wasted on regions the camera has already ruled out. We pair it with a cheap way to fold in late detections, and test whether the combination keeps track of the object through long occlusions within a real-time budget.

Core Claim

Our claim is that during occlusion the dominant source of particle failure is not bad motion models. It is that blind propagation keeps placing particles in visible free space, where the depth image kills them. This repeated loss compounds over the occlusion until one particle carries all the weight, the failure van Leeuwen et al. describe as particle degeneracy. An exact, closed-form proposal along the viewing ray removes that loss at no extra sampling cost. It should keep the effective number of particles near its nominal value through occlusions that destroy a bootstrap filter given the same compute. The bootstrap filter is the standard particle filter that propagates blindly and only reweights afterwards.

Formally, let a particle's motion prediction place the object center at pixel $(u,v)$ with depth distributed as $\mathcal{N}(m, s^2)$ along that pixel's ray. Let $D(u,v)$ be the observed depth there, and let $\delta$ be a margin set by the object's radius. If the object is unseen, the proposal draws $z$ from that Gaussian restricted to $z > D(u,v)+\delta$. The weight update is

$$ w_t \propto w_{t-1}\left[1 - \Phi\!\left(\frac{D(u,v)+\delta - m}{s}\right)\right], $$

where $\Phi$ is the standard normal cumulative distribution. This is the optimal importance density for a binary "object is hidden" likelihood, because the optimal proposal is prior times likelihood, and a 0/1 likelihood simply truncates the prior. If the detector can also miss a visible object with probability $\epsilon$, the likelihood becomes 1 behind the surface and $\epsilon$ in front of it. The optimal proposal is then a two-component mixture of the truncated and untruncated Gaussians, with mixing weights given in closed form by the same $\Phi$ term. Pixel coordinates and rotation are still sampled from the motion prior; only depth is treated exactly, in the spirit of Rao-Blackwellization (handling one part of the state analytically so that sampling noise is confined to the rest).

The prediction is quantitative and falsifiable. The advantage over a bootstrap filter at equal wall-clock time should grow with two quantities: the ratio of the prediction's depth spread to the occluder's depth extent, and the occlusion's duration in control steps. A bootstrap filter's survivor fraction per step is roughly the bracketed term above, and those survival rates multiply across steps between resamplings. For wide, deep occluders the bracketed term is near 1 and the gain should vanish. We state this regime boundary up front because it is exactly where a skeptic would push.

Relation to Prior Work

Three lines of work motivate the gap. PoseRBPF splits 6D pose into translation particles and a learned codebook distribution over rotations, which handles symmetric objects well. Its translation particles, however, move by a blind constant-velocity prior, and it runs at about 20 fps with 200 particles. We keep its rotation treatment and replace the translation proposal. Temporally Consistent Object 6D Pose Estimation for Robot Control models monocular depth uncertainty as an ellipsoid stretched along the viewing ray inside a factor graph, a graph-based least-squares smoother. That is the observation that makes ray coordinates natural for us. Its estimate is unimodal, and during long dropouts it has only kinematic integration to rely on. ROFT reaches 96.2 fps by fusing optical-flow velocity with delayed network detections in an unscented Kalman filter, which also keeps a single Gaussian estimate. Optical flow disappears when the object is fully hidden, which is precisely our regime.

Two works supply technique. PART masks occluded pixels out of the depth likelihood when tracking the robot's own arm. It also handles 50 ms-delayed images by rewinding a buffer and replaying encoder commands. Our proposal differs from PART's masking in a specific way. PART uses occlusion to avoid corrupting the likelihood; we use the visible surface to shape where particles are drawn, and we track external objects that have no encoders to replay. The mixture proposals of Thrun et al. blend motion samples with samples inverted from the current sensor reading. We reuse this idea for re-emergence: a late detection injects a fraction of particles around itself.

Two works constrain the design. Twisted SMC steers particles toward future constraints using look-ahead functions. It shows that strong steering pays off only if the proposal's inner sampling is cheap, and it runs offline at seconds per trajectory. That is why we refuse any learned or nested proposal and insist on a closed form. The Unscented Particle Filter approximates the optimal proposal with a per-particle Gaussian filter. For a truncation likelihood that approximation is unnecessary, and it is too expensive at control rate.

On the control side, Xu et al. set out the conflict between slow perception and 100–1000 Hz control loops. RTC and REMAC make action-chunked policies tolerate inference delay. Both assume the target is visible; we supply what they lack, a belief over a hidden target. CMSMC embeds a learned trajectory predictor as the proposal of a mixture particle filter for occluded vehicles. That is the closest "informed proposal under occlusion" work in the file. Its proposal is informed by behavior forecasts, not by the current measurement. It works in 2D road coordinates at traffic rates and has no delayed-measurement handling. We found no paper in the file that turns the depth image into an occlusion-constrained proposal.

Research Program

The system has three parts, each kept within an O(N) per-step cost for N particles.

The first part is the occlusion-truncated proposal above, run every control cycle. Visibility is decided by whether the fast segmentation, or a rendered silhouette test against the current depth frame, finds the object. Partial occlusion, where some of the object's model points are visible, falls back to the standard rendered-depth likelihood with occluded pixels masked as in PART. The truncation applies only once the visible fraction drops below a threshold. A sweep over that threshold is part of the study.

The second part handles late detections by weighting ancestors. Each particle stores a short ring buffer of its own past states along its ancestry. When a detection timestamped $d$ steps ago arrives, current particles are reweighted by the likelihood of that detection evaluated at their ancestor's state at that time. This is a fixed-lag particle-smoothing view of the out-of-sequence measurement. It costs O(N) rather than PART's O(Nd) rewind-and-replay. The known risk is path degeneracy: after many resamplings, many current particles share the same ancestor, so the lagged states lose diversity. We will measure ancestor diversity as a function of $d$ and of the resampling threshold, and compare directly against full replay. The third part is a gated re-emergence mixture: a fraction of particles is drawn around the detection when it falls outside the cloud's predictive region, so a valid re-detection is never discarded as an outlier.

The deliverable is a GPU tracker whose translation update for a few thousand particles fits within a 1 ms slice. The full cycle must fit within a 10 ms (100 Hz) budget, and we will report 90th- and 99th-percentile latency. Its output feeds two downstream consumers: a simple intercept controller, and an RTC-style chunked flow policy that receives a particle summary (mixture means and covariances) in place of raw crops of the target.

Baselines

The naive reference is last-pose hold plus constant-velocity extrapolation during occlusion, with the detector resuming on re-emergence. This is what a policy fed raw detections effectively sees. Any method must beat it on error during occlusion and on re-acquisition time, or the problem is not real on our data.

The strongest existing real-time competitor is ROFT. On Fast-YCB the file reports an average angular error of 12.6° for ROFT, against 42.9° for PoseRBPF and 34.1° for se(3)-TrackNet. These are unoccluded-regime numbers from the ROFT paper. We run ROFT's released pipeline on the same frames, with the same delayed detector (identical network and latency injection) and the same hardware. Beating it means matching its 12.6° on clean Fast-YCB within noise, since our method should not hurt the visible case. It also means a lower error and re-acquisition time on our occluded variants, at a 90th-percentile cycle time no worse than ROFT's 10.4 ms.

The second strong competitor is the factor-graph smoother of Zorina et al.. It reports recall 0.65 and precision 0.81 on HOPE-Video, against CosyPose's 0.39 and 0.57. We feed it the same detections and tune its gating threshold on the same validation split we use for our own thresholds. PoseRBPF is included as the particle filter with a blind translation proposal.

The decisive ablation is a bootstrap particle filter with the identical visibility likelihood, motion model, and rotation codebook, given the same GPU wall-clock budget. Because it is cheaper per particle, it is allowed more particles. Only the proposal differs, so this isolates the core ingredient. Beating it means higher effective sample size and lower error in the predicted regime (narrow occluders, long occlusions), not merely on average. Further ablations replace ancestor weighting with PART-style full replay, or simply drop late detections, and remove the re-emergence mixture.

Why not just…?

Why not just run a bootstrap filter with ten times more particles on a GPU? This is the most serious threat, and we cannot argue it away. If occluders are wide in depth, blind particles rarely land in visible free space, and extra particles buy back most of what truncation offers. The matched-compute ablation, combined with a sweep over occluder depth extent and occlusion duration, settles it. We commit to reporting the regime where the gap closes. If the gap closes everywhere on realistic scenes, the contribution reduces to the late-detection handling, and we will say so.

Why not just use an unscented Kalman filter or factor graph with occlusion gating? A Gaussian cannot represent "the object either stopped behind the box or kept going and is about to emerge on the right." A Gaussian forced to fit that situation centers its mean between the two possibilities. Mixture SMC beating extended and unscented Kalman filters under nonlinear, occluded motion in CMSMC is indirect evidence for this. Our occluded sequences deliberately include objects that stop, reverse, or turn behind the occluder. On constant-velocity occlusions we expect ROFT to remain competitive, and we will report that split separately.

Why not let the policy handle occlusion itself, for instance by training a Diffusion Policy or REMAC with occlusion augmentation? Diffusion Policy conditions on a short observation history. REMAC shows that open-loop execution against stale perception costs success: synchronous execution reached 67.0% against REMAC's 81.2% on Grasp-Hard in REMAC. Neither keeps a belief over a hidden target. We test this directly by training the same chunked policy two ways: on raw crops with random synthetic occlusion augmentation, and on the particle summary. Both get equal data and equal training steps.

Why not just mask occluded pixels in the likelihood, as PART does? That is our "likelihood-only" ablation. It prevents occluders from pulling the estimate, but particles still land in visible free space and die. The predicted difference shows up specifically in effective sample size and track-loss rate during full occlusion.

Proposed Experiments

The tracking study uses Fast-YCB, whose photorealistic synthetic rendering lets us insert occluders. We build an occluded variant by passing rendered bars and boxes of controlled width and depth extent in front of the moving objects, with occlusion durations from 5 to 50 control steps. We report angular and translational error during and after occlusion, re-acquisition time, the fraction of tracks lost, and effective sample size over time. YCB-Video tests generalization to real clutter; the file reports 49.86 average recall for Ponimatkin et al. and 42.95 for MegaPose. HOPE-Video provides the comparison against Zorina et al. in recall and precision. HO-3D covers hand occlusion during manipulation; we found no reported result on Lacuna for HO-3D, so we establish our own baselines there. Delayed detections are injected at 100, 200, and 300 ms, matching the latency range RTC targets.

The closed-loop study runs on a simulated and then physical Franka arm. An object slides along a conveyor behind an occluder that may stop or reverse it, and the arm must intercept it on emergence. We measure success rate and a jerk-based smoothness measure. Both downstream consumers are tested: the intercept controller and the chunked flow policy with RTC-style inpainting. Each is fed either ROFT's output, the bootstrap filter's output, or ours. The decisive checks are three. Effective sample size must follow the predicted dependence on occluder depth extent and duration, including vanishing gains for wide occluders. Ancestor weighting must match full replay in accuracy up to some delay, beyond which path degeneracy breaks it; we will find and report that delay. And closed-loop interception must improve over ROFT on the stop-or-reverse scenarios while every cycle stays within its 10 ms budget.