When a robot reaches for an object, its own arm and fingers block the cameras at the moment vision matters most. Current systems respond in one of two ways. Some keep a remembered belief that the object is still where it was last seen. Others stop the motion when failure looks likely. Neither can answer the question that decides whether the task recovers: if the gripper just knocked the object while nobody could see it, where is it now? Long-term visual trackers already have a mechanism for lost targets. They switch from following the target locally to searching the whole image. But they were built for video in which disappearance is unexplained, so they search blindly and are fooled by look-alike objects. A robot is in a better position than a video tracker. It knows its own joint angles, its link shapes, and its camera poses, so it can compute exactly which parts of the workspace it is currently hiding.
This note proposes a tracker for manipulation that uses this self-knowledge in two ways. First, it separates expected disappearance (the target sits inside the robot's own computed occlusion shadow and nothing has touched it) from unexpected disappearance (a contact event happened while the target was hidden). In the first case it holds its belief. In the second it starts a search. Second, when it does search, it restricts candidates to places the object could physically have reached, and it ranks them by whether each place is visible now or is hidden behind the arm. If the most likely places are hidden, the recovery action is the arm movement that uncovers them most cheaply. The contact detector runs on a fast, low-resolution video pathway, and appearance verification runs on a slow, high-resolution pathway, following the asymmetric design of SlowFast Networks for Video Recognition.
Core Claim
The central claim is that, in manipulation, the switch from local tracking to global search should be triggered by contact evidence combined with computed self-occlusion, not by a drop in tracker confidence. A confidence threshold cannot tell "my arm is in front of the cup" from "the cup is gone," because both produce low confidence. The consequence is that a confidence-gated tracker either searches constantly during normal grasps, producing false re-acquisitions on distractors, or never searches when it should. We claim that a self-occlusion-aware gate, together with a search restricted to reachable and shadowed regions, will re-acquire displaced objects faster and with fewer distractor errors than confidence-gated long-term tracking. It will also avoid the blind execution of memory-based planners when a hidden object has moved.
Where This Sits
The re-detection machinery comes from long-term tracking. 'Skimming-Perusal' Tracking pairs a local tracker and a learned verifier ("perusal") with a lightweight full-image scanner ("skimming"). It declares the target lost when verifier confidence falls below 0.65. It reports an F-score of 0.616 on the VOT2018LT long-term benchmark at 25.7 frames per second. We reuse its two-state structure and its verifier idea, but replace both the trigger and the search space. Self-Paced Learning for Long-Term Tracking argues that motion extrapolation such as a Kalman filter diverges during long disappearance, so recovery must be appearance-based search. This constrains our design: we do not try to predict where a knocked object went. We only bound where it could be. The Long-Term Visual Object Tracking Benchmark documents how short-term trackers corrupt their templates on background during absence. This is why our slow pathway freezes template updates whenever the target is inside the shadow.
The manipulation work motivates the gap. RoboStream keeps object permanence through a causal graph of action-triggered state changes. It reaches 88.9% on a "Hide and Restore" task where reactive models score 0%, but its belief is symbolic and is never checked against perception if something unmodelled moves the object. ContactGuard forecasts latent visual states one second ahead to predict pre-contact failures, with an area under the ROC curve of 0.992 on its cup task. It only aborts and offers no way to re-acquire the target. It also reports that adding joint proprioception hurt failure prediction, because the model used it as a shortcut. We take that finding seriously, so proprioception enters our system only through fixed geometry (the shadow computation) and an ablated torque-residual cue, not as a free input to a learned predictor. DECO shows that tactile input recovers performance under finger occlusion, with a 20% gain on contact-rich tasks. Touch is silent once the object leaves the hand, though, which is exactly our regime. RVT re-renders point clouds from virtual viewpoints. That cannot show surfaces that no physical camera observed, and it is the reason we reason about the unobserved volume explicitly.
The half-space coverage literature enters as a geometric lens rather than as hardware. Liu et al. and Zeng et al. treat coverage as something computable from surface placement: a reflecting surface serves one half of space and leaves a known dead zone. We treat each camera the same way. Its visible set, minus the shadow cast by the arm's current configuration, is computable every frame, and the recovery question becomes how to cover the dead zone that holds probability mass. We deliberately do not propose RF metasurface sensing of the object. The STAR-RIS prototype in Design and Prototyping of Filtering Active STAR-RIS operates near 2.6 GHz, where the wavelength is roughly 11.5 cm. That is too coarse to localize tabletop objects at grasping precision. Work that joins STAR-RIS with robot tracking, such as STAR-RIS-Aided ISCC for Internet of Robotic Things, tracks whole robots by radar and assumes they are stationary.
Research Program
The system has three parts. The first is the shadow map. At each control step we take the arm and hand meshes at the current joint configuration and the calibrated camera poses. We mark every workspace voxel $x$ as visible if at least one camera ray to $x$ is unobstructed by the robot, and as shadowed otherwise. The shadowed set $H_t$ changes as the arm moves and costs one batched ray-mesh intersection per frame. When the last confirmed target position lies inside $H_t$ and no contact event has fired, the tracker is in a held state. It freezes the template, keeps the last position, and does not search.
The second part is the two-rate perception stack. The fast pathway runs at camera rate (30–60 Hz) on low-resolution crops around the end-effector, with few channels. Following the capacity argument of Hao Li et al., the low capacity is intended to force it to model motion rather than appearance. It is trained to fire a binary "unplanned contact or object motion" event, using scripted-knock data plus the failure episodes that ContactGuard-style datasets contain. The slow pathway runs at a few Hz on full-resolution views. It maintains the target appearance embedding and scores candidates in the manner of the perusal verifier. An ablated variant adds joint-torque residuals to the event detector, so we can measure whether proprioception helps event detection even though it hurt failure forecasting in ContactGuard.
The third part is constrained search and uncovering. A contact event opens the search state. The search prior over voxels is
$$p(x) \propto \mathbb{1}\left[\lVert x - x_{\text{last}} \rVert \le r\right] \cdot s(x),$$
where $x_{\text{last}}$ is the last confirmed position, and $r$ bounds displacement given the end-effector speed at contact plus a fall-off-edge region. The support term $s(x)$ is 1 on the table and supporting surfaces and small elsewhere. Visible candidates are scored by the slow verifier. If the verified mass among visible candidates is below threshold, the remaining mass lies in $H_t$. The robot then picks, from a small library of retract and rotate motions, the motion whose predicted shadow change exposes the most prior mass per unit of motion time. It then re-scores. Only after this constrained search fails does the system fall back to a full-scene skim.
Baselines
The naive reference is a short-term tracker with no lost-target logic, run on the same camera streams. It shows the size of the performance cliff in manipulation and should fail almost every knock-while-hidden trial.
The strongest existing re-detection method is Skimming-Perusal tracking (SPLT). It reports F-score 0.616 and MaxGM 0.622 on OxUvA, and is the most direct competitor because it already implements the track-versus-search switch. We run it on identical multi-view streams. Its verifier is fine-tuned on the same object crops as ours, and its confidence threshold is tuned on the same validation episodes with the same number of trials that we spend tuning our event threshold. As a second variant, we let it skim with the same compute budget per second as our full system. Beating it means a lower median time-to-reacquire, a lower false re-acquisition rate when a same-looking distractor is present, and a lower rate of spurious searches during undisturbed grasps, all at matched latency. We also check that our component tracker does not fall below SPLT's reported 0.616 F-score on VOT2018LT, where no self-occlusion geometry is available. That check tells us whether we paid for manipulation gains with generic tracking quality.
The strongest memory-based competitor is a RoboStream-style held belief, which never searches and always trusts the last position until the robot sees evidence against it. On undisturbed hide-and-restore it should match us, since RoboStream reports 88.9% there. Beating it means higher task success specifically when the hidden object was displaced. The abort-only competitor is a ContactGuard-style monitor that stops at predicted failure. We compare task completion, not abort accuracy, since aborting is where ContactGuard stops. For DECO-style tactile fallback, we report the fraction of displacements it recovers with no vision at all.
Three ablations isolate our ingredients. One replaces the self-occlusion gate with a confidence threshold. One replaces the reachability-plus-shadow prior with full-frame search. One replaces the uncovering motion with random retraction.
Why not just…?
Why not just add more cameras or an active camera arm? This is the most serious alternative for accurate localization in cluttered cells, and we treat it as a covariate rather than dismissing it. ContactGuard's setup still suffers terminal self-occlusion with three cameras, and wrist cameras go blind as the gripper closes. So extra views shrink the shadow but do not remove it. Our method still needs something to tell expected from unexpected disappearance within whatever shadow remains. We will run every condition with one, two and three external cameras. If the gap between our system and SPLT disappears at three cameras, the honest conclusion is that the method matters only for sparse-camera setups, and we will report it that way.
Why not just use SPLT's confidence gate, since it already works? The confidence gate has no way to know the arm caused the low score. We expect it to either trigger searches during every normal grasp or be tuned so conservatively that it misses real losses. The gate ablation, which uses our own verifier and search but a confidence trigger, directly tests whether the self-occlusion gate is the ingredient that matters. If it performs as well, the contribution collapses to the constrained prior alone, and we would say so.
Why not just keep a memory of where the object was, as RoboStream does, or extrapolate with a Kalman filter? When nothing disturbs the hidden object, memory is correct and cheaper, and our held state reproduces exactly that behaviour. Once the arm has knocked the object, the stored position is simply wrong. Self-Paced Learning for Long-Term Tracking argues that motion extrapolation diverges without measurements. The displaced-while-hidden condition separates these systems, and the undisturbed condition checks that we lose nothing to them.
Why not just train an end-to-end policy with a longer visual history? A large policy could learn recovery implicitly, but it gives no explicit lost-target state to evaluate. ContactGuard's shortcut finding also warns that learned models misuse proprioception. We include a history-conditioned policy trained on the same recovery demonstrations as a baseline. If it matches our re-acquisition rates, the explicit structure is unnecessary for this setting.
Why not just sense through the arm with RF, as full-space metasurfaces suggest? At the roughly 11.5 cm wavelength of the 2.6 GHz prototype, the resolution is far coarser than the object displacements we need to resolve. No paper in our search reports metasurface localization of tabletop objects. We therefore leave RF out of scope rather than build a weak version of it.
Proposed Experiments
No existing benchmark measures re-detection of objects displaced during self-occlusion. No reported result was found on Lacuna for this setting. We will therefore build a protocol with two parts. In simulation, RLBench-style tabletop tasks are run with scripted gripper-object collisions that occur only while the object lies in the computed shadow. The collisions vary displacement magnitude, the number of identical distractors, and camera count. On a real arm, the same knock-while-hidden protocol is applied to cup and box pick-and-place, mirroring the ContactGuard tasks. The event detector's generality is checked on held-out occluded contact episodes from DECO-50. The standalone tracker is sanity-checked on VOT2018LT and on the heavy-occlusion HOOT benchmark.
Metrics are median time-to-reacquire, 3D position error at re-acquisition, false re-acquisition rate on distractors, spurious search rate during undisturbed grasps, and downstream task success after recovery.
Three checks decide the claim. First, with two identical distractors placed outside the reachable radius, our system should almost never lock onto a distractor, while full-frame skimming should. Second, on undisturbed grasps our spurious search rate should be near zero, while SPLT's confidence gate should fire on most terminal approaches. Third, on displaced-while-hidden trials, task success should clearly exceed held-belief memory, with the advantage shrinking but not vanishing as external cameras are added. If the second check fails, the event detector is the weak link, and the torque-residual ablation shows whether proprioception can repair it.
References
- SlowFast Networks for Video Recognition
- 'Skimming-Perusal' Tracking: A Framework for Real-Time and Robust Long-Term Tracking
- Self-Paced Learning for Long-Term Tracking
- Long-Term Visual Object Tracking Benchmark
- RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- RVT: Robotic View Transformer for 3D Object Manipulation
- STAR: Simultaneous Transmission and Reflection for 360° Coverage by Intelligent Surfaces
- Enabling Panoramic Full-Angle Reflection Via Aerial Intelligent Reflecting Surface
- Design and Prototyping of Filtering Active STAR-RIS with Adjustable Power Splitting
- STAR-RIS-Aided Integrated Sensing, Computing, and Communication for Internet of Robotic Things
- A Novel Spatio-Temporal 3D Convolutional Encoder-Decoder Network for Dynamic Saliency Prediction
- HOOT: Heavy Occlusions in Object Tracking Benchmark