When a person wearing smart glasses picks up a mug, or a robot's wrist camera closes in on a part, the object disappears behind fingers at exactly the moment it matters most. During that interval the object's motion is not random. Either it sits still in the world, or it rides along with the hand that is gripping it. Both the hand and the camera are tracked well by other sensors: hand trackers and head-pose tracking on glasses, joint encoders on a robot. We propose to track hidden objects by keeping a probability distribution over two things. The first is which of these regimes the object is in: resting, held by the left hand, or held by the right hand or gripper. The second is where the object is, computed exactly given that regime. When vision goes dark, the tracker then carries the object along with the right hand. When vision returns, the tracker has a calibrated guess for where to look again.
The tool is a Rao-Blackwellized particle filter (RBPF). A standard particle filter represents uncertainty by a cloud of random guesses. An RBPF samples only the hard, low-dimensional part of the state and computes the rest in closed form for each guess, which needs far fewer guesses. Here the sampled part is the discrete contact mode together with the object's rough position. The closed-form part is the object's offset relative to whatever anchors it, plus its rotation. The novelty is not using an RBPF for object pose, which PoseRBPF already does from fixed cameras. The novelty is making the hand or the camera the motion model and treating "who is holding the object" as an uncertain hidden variable that is inferred, rather than decided by a contact heuristic.
Core Claim
During hand-induced occlusion in first-person video, most of the error of current trackers comes from getting the anchor wrong, not from appearance modeling. The anchor question is whether the object moved with the hand or stayed put. A filter that keeps a posterior over the anchor, and computes the anchored offset in closed form, should keep pose error bounded through full occlusions. It should also reacquire objects faster than deterministic pipelines, and give calibrated uncertainty that a downstream controller can use. The claim is falsifiable against one specific competitor, a deterministic "attach to hand on contact" heuristic. If that heuristic matches the filter, mode uncertainty is not worth modeling, and we will say so.
How the Filter Works
At time $t$ the hidden state is the contact mode $c_t$ (resting, left-held, right-held), the object's position, and its rotation $R_t$. Let $A_t(c_t)$ be the pose of the anchor implied by the mode. This is the identity world frame when resting, or the tracked hand or gripper pose when held. The object pose is $A_t(c_t)\,\delta_t$, where $\delta_t$ is a slowly drifting offset. The posterior factorizes as
$$p(c_{1:t}, \delta_t, R_t \mid y_{1:t}) = p(\delta_t, R_t \mid c_{1:t}, y_{1:t})\; p(c_{1:t} \mid y_{1:t}),$$
where $y_{1:t}$ are the image observations. Particles sample mode histories $c_{1:t}$ and are propagated using the known anchor motion. This is the FastSLAM pattern analyzed in the marginalized particle filter, where odometry drives the proposal and sensing only reweights. Given a mode history, the translational offset is tracked by a small Kalman filter: the observed 3D object center is a linear function of the offset once the anchor pose is fixed. Rotation is scored over a discrete grid with a learned embedding codebook, reusing PoseRBPF's rotation marginalization.
The likelihood is visibility-aware. Each frame we render where each particle predicts the object to be, and compare that against hand masks and depth.
- A predicted location that falls behind the hand gets an uninformative likelihood.
- A predicted location in clearly visible free space that shows no detection gets down-weighted. This "negative information" is what lets the filter discover that an object was left behind rather than carried.
Mode transitions use a prior that rises when the hand is near the object and nearly stationary relative to it. A contact detector can feed this prior, but it never makes the decision outright.
This design also addresses a tension in the file. ContactGuard, which learns a latent world model to predict grasp failure, found that feeding joint states into that model hurt it in low-data regimes because the model used them as a shortcut. Image-to-sphere policies, by contrast, gained large improvements by using wrist orientation as an exact geometric correction. Our filter uses proprioception and hand pose only geometrically, as the anchor motion, so no learned component can latch onto them.
Relation to Prior Work
Two lines of work supply the method. PoseRBPF samples translation and marginalizes rotation over a codebook. It reports real-time tracking at "up to 20fps for RGB tracking with 200 particles" on YCB Video and T-LESS. Its constant-velocity motion prior was only ever exercised with third-person cameras. It has no notion of an object being carried, so under a held occlusion its particles diffuse or freeze in place. Goal inference with RBPFs shows the same pattern of sampling a discrete latent and computing kinematics with a Kalman filter, but only in 2D synthetic navigation.
Three systems motivate the gap. EgoTrack3D tracks 3D boxes in Aria glasses video. It uses point tracking, Hungarian matching, and a short SAM2 track triggered when a hand-object detector fires. It has no uncertainty, and it fragments tracks under long occlusion, as its own reliance on voxel merging for recovery suggests. DexYCB quantifies the problem: pose accuracy drops "from approximately 56% to 41% when an object is grasped." VFAS-Grasp goes open-loop in the final inch, and ContactGuard can only abort. Neither keeps a belief that would support recovery.
A fourth system constrains the design. The tactile-visual Bayesian filter shows recursive filtering can localize contact once vision is fully blocked. We treat touch as future work and limit ourselves to vision plus anchor motion, so the effect of the mode variable is not confounded with a new sensor.
Research Program
The work has three stages.
The first stage builds the filter on Aria Digital Twin. Aria Digital Twin (ADT) provides object models, device pose, 3D human pose, and motion-capture ground truth accurate to "6.83 mm and a rotation error of 1.29°". We train per-object rotation codebooks as PoseRBPF does. We run the filter twice: once with estimated hand and head pose, which is the realistic setting, and once with ground-truth hand pose. The ground-truth run is an oracle that separates anchor-tracking error from filter error.
The second stage carries the filter to DexYCB. There we treat single camera views independently to test whether the gains hold where grasp-induced degradation is already documented.
The third stage ports the filter to a wrist-camera robot. The gripper is the anchor and its pose comes from joint encoders. We test whether the belief that survives the blind final inch, and survives an induced slip, supports re-grasping. This is something no monitor in the file does.
Baselines
The naive reference keeps the last confident pose fixed in the world frame and does nothing else. It is correct whenever the object is resting and wrong whenever it is carried, so it bounds how much carried-object handling matters in each dataset.
The strongest existing system is EgoTrack3D. It reports "a 63.01% average PCL" on ADT, where PCL is the percentage of correct locations, "an 11% relative improvement over the strongest baseline (Boxer)." We will run it on our exact ADT split from released code where possible, using the same upstream detector and camera poses as our filter. Beating it means higher average PCL on the full split and a clear margin on the subset of frames inside hand-occlusion intervals, which we will extract from ADT's segmentation ground truth.
The strongest probabilistic competitor is PoseRBPF made egocentric. It runs in the world frame using the device pose, with its standard motion prior and no modes, and gets the same codebooks and particle count. On DexYCB the relevant single-frame reference is CosyPose's reported "57.4% Average Recall (AR)". No reported result was found on Lacuna for PoseRBPF on ADT or DexYCB, nor numeric YCB Video ADD values for it. We will run it ourselves.
The key ablation is a deterministic mode switcher. It is our filter with a single particle per object whose mode is set by the contact detector. It isolates exactly the value of keeping uncertainty over the anchor. Two further ablations remove the negative-information term and replace Rao-Blackwellization with a full six-dimensional particle filter at matched wall-clock compute.
All methods share detectors, pose sources, and a fixed tuning budget on a held-out ADT validation set.
Why not just…?
Why not just compensate for camera motion and run PoseRBPF in the world frame? For resting objects this is close to what we do, and we expect it to match us there. Its failure is predictable: when the hand carries a hidden object, a world-frame motion prior can only spread uncertainty outward, so particles leak into empty space. The occlusion-interval comparison on ADT, broken down by whether ground truth shows the object moving, will show whether this gap is real. If the gap is small, carried occlusions are rare in the data and our claim is weakened.
Why not just glue the object to the hand when a contact detector fires? This is the real threat. It is cheap, and if contact detection is reliable it captures most of the benefit. Our bet is that hand-object detectors are least reliable exactly when the hand covers the object. We also bet that wrong attach and release decisions are unrecoverable without a distribution over modes. The deterministic-mode ablation tests this directly. We will report how its error grows as we artificially corrupt detector output. If it matches the full filter at realistic detector quality, we will recommend it.
Why not just let an end-to-end policy with memory handle occlusion? The evidence in the file cuts against this in low-data regimes. A standard diffusion policy reached only 15% on occluded 3D-pipe disassembly versus 75% for the image-to-sphere policy. Even that stronger policy has no explicit object state with which to recover after an abort. In the robot stage we will condition the same policy on our belief versus on raw history, and measure re-grasp success.
Why not just add cameras? AV-ALOHA shows that an active camera arm helps on occluded tasks, for example Thread Needle rising from 30% to 52%. It also shows that the larger action space hurts simple tasks. On glasses and mobile robots, extra cameras are not available at all.
Proposed Experiments
On ADT we report four measures: average PCL; translation and rotation error inside occlusion intervals as a function of interval length; frames until reacquisition after reappearance; and calibration, meaning negative log-likelihood and the coverage of 90% credible regions. The decisive check is error versus occlusion length. Our filter should stay near its visible-phase error for carried objects, while the world-frame and naive baselines should grow roughly with distance traveled by the hand. Running the filter with oracle hand pose tells us how much of any remaining gap is due to hand tracking.
On DexYCB we test whether the filter closes part of the documented 56%-to-41% grasp penalty.
On a wrist-camera Franka setup modeled on VFAS-Grasp, we use its handover setting as the reference point, where it reports an "81.25% success rate". We measure grasp success and re-grasp success after induced slips, with and without the belief passed to the grasp sampler.
A negative result on the deterministic-mode ablation, at realistic detector quality, would retire the central claim.
References
- PoseRBPF: A Rao-Blackwellized Particle Filter for 6-D Object Pose Tracking
- The marginalized particle filter – analysis, applications and generalizations
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 3D Equivariant Visuomotor Policy Learning via Spherical Projection
- Goal inference with Rao-Blackwellized Particle Filters
- EgoTrack3D: A Modular Framework for Egocentric 3D Object Tracking
- DexYCB: A Benchmark for Capturing Hand Grasping of Objects
- VFAS-Grasp: Closed Loop Grasping with Visual Feedback and Adaptive Sampling
- Localizing the Object Contact through Matching Tactile Features with Visual Map
- Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation