When a person teleoperates a robot to insert a peg, seat a dish in a rack, or turn a valve, the hardest moment comes at the end. The robot's own wrist and fingers move between the camera and the contact point, so the operator loses sight of the part of the scene that matters most, just when the tolerances are tightest. This note proposes a copilot that predicts that moment from the robot's own geometry and takes over only the short, hidden final segment. Before the hand covers the target, it smoothly shifts control authority from the human to a policy that feels its way using touch. Because the copilot only has to learn that short terminal segment, it can be trained with reinforcement learning from a single success/failure signal at the end of the task. Over long tasks, that kind of signal usually makes learning hopeless.
The opportunity comes from a coincidence that prior work has treated as three separate problems. Self-occlusion is where vision-based policies fail. It is also where human operators hesitate and over-correct. And it is the only part of a long manipulation task where a sparse terminal reward is close enough in time to the actions that caused it. A single visibility-predicting gate addresses all three. It lowers the operator's workload where the workload peaks. It routes control to tactile sensing where cameras are useless. And it shrinks the reinforcement-learning horizon to a window short enough that sparse reward becomes workable for a high-dimensional hand.
Core Claim
We claim that an assistance gate driven by predicted occlusion of the contact zone beats both fixed-blend copilots and uncertainty-triggered handover for contact-rich teleoperation. The occlusion is computed by rendering the robot's own forward-kinematic mesh at its near-future configuration. We expect three gains: higher success, lower operator workload, and much better sample efficiency when the copilot is trained from sparse reward. The key word is predicted. Uncertainty-based handover, as in PATO, reacts after the policy is already confused. A geometric forecast gives lead time, so the transfer of authority happens while the operator can still see. This avoids the re-orientation cost that the cognitive-load literature identifies as the main price of switching control.
Concretely, the executed action is
$$a_t = a^h_t + \alpha_t\, a^r_t, \qquad \alpha_t = g\big(\hat v_{t:t+H}\big),$$
where $a^h_t$ is the operator's command, $a^r_t$ is a residual action from the copilot, $\hat v_{t:t+H}$ is the predicted fraction of the contact zone hidden from the operator's camera over the next $H$ steps, and $g$ is a monotone map from that predicted occlusion to copilot authority $\alpha_t\in[0,1]$. The residual copilot is trained to maximize sparse task success while staying out of the way:
$$\max_\pi \; \mathbb{E}\Big[\mathbf{1}[\text{success}] - \lambda \sum_t \lVert \alpha_t a^r_t \rVert\Big],$$
where $\lambda$ trades success against intervention magnitude. This is the residual formulation of Residual Policy Learning for Shared Autonomy, which trained additive copilots with constrained policy optimization on fully observed low-dimensional state in Lunar Lander and a drone reacher. Our version differs in two ways. The copilot acts on a high-dimensional hand under occlusion, and its authority is modulated by geometry instead of being always on.
Research Program
The first component is the visibility forecaster. At each control step, we extrapolate the robot's configuration over a short horizon (roughly 0.5 s). The extrapolation uses the operator's current command and the copilot's action chunk. We then rasterize the robot's link meshes at low resolution from the operator-display camera. We measure how much of the contact zone those links cover. The contact zone is the target region (socket, rack slot, valve stem), localized while it was still visible and held in memory afterward. This is similar in spirit to the memory effect in Chen et al.'s drawing avatar, where a second arm moved a camera to recover hidden canvas regions. We keep the camera fixed and instead move authority.
The forecaster is pure geometry and should run in milliseconds. This matters for two reasons. ActiveVLA reports 0.53 s per inference step for active viewpoint selection versus 0.26 s for static views. And delays of that size in a teleoperation loop make operators over-steer.
The second component is the occlusion-phase copilot. It is a visuo-tactile residual policy for an arm with a multi-fingered hand: an Allegro-class hand on a 7-DoF arm, matching the 23-DoF Kuka-Allegro setting of ADEPT. The operator commands wrist pose and a coarse grasp aperture through a 6-DoF input device. The copilot outputs residual wrist and finger-joint actions. Its tactile pathway borrows the design of DECO. DECO injects 1,062 tactile channels into a frozen vision-action diffusion transformer through a small cross-attention adapter. It reports a 20% absolute gain on contact-rich tasks over vision alone, which is our evidence that touch carries the information vision loses under occlusion. Because diffusion sampling adds latency, we distill the copilot to a single-pass network for closed-loop use. The diffusion-based DiSCo is kept as a baseline.
The third component is sparse-reward training in simulation. We build simulated versions of the Functional Manipulation Benchmark asymmetric peg insertion and of ADEPT's dish-rack task. We pair them with simulated operators whose command noise, lag, and hesitation grow as their view of the contact zone shrinks. The idea of corrupting operator commands follows DiSCo's evaluation. The new element is that the corruption is tied to occlusion. The copilot is initialized by behavior cloning and then fine-tuned with conservative policy-gradient updates using only the terminal success bit.
We use ADEPT's warm-up recipe, which trains the value function with the actor frozen before any policy update. ADEPT found that fine-tuning directly from a pretrained policy collapses on sparse tasks, so this design choice is constrained by that negative result. The hypothesis specific to this proposal concerns the gate: because it confines nonzero copilot authority to the final occluded segment, the effective credit-assignment horizon shrinks from the whole episode to a few seconds. Sample efficiency should therefore improve by a large factor over an ungated residual copilot trained on the same reward. We deliberately do not use a learned visual progress critic such as VLAC as the dense reward during the occluded phase. That critic assumes progress is visible and monotone. Neither assumption holds while the hand hides the contact or the operator hesitates. We include it as a baseline instead.
The fourth component is a human study on real hardware. Participants teleoperate occluded insertion and rack-placement tasks under each assistance condition, with a fixed camera layout.
Baselines
The naive reference point is unassisted teleoperation with the same input device and the same fixed camera. It sets the floor for success, completion time, and workload. For calibration, DiSCo reports 45% success for no copilot on its Block Choice task and 28% on Drawers. Those tasks have no deliberate occlusion, and we expect our floor to be lower.
The strongest existing shared-autonomy method is DiSCo. It is a diffusion copilot that seeds and inpaints the operator's recent actions into its denoising, with fixed weights trading operator intent against expert adherence and responsiveness. DiSCo reports 80% success on Block Choice and 42% on Drawers under corrupted inputs. We run it on our tasks with the same demonstration data. We give it our tactile adapter so that it is not handicapped on sensing, and we tune its blending weights over the same grid and number of pilot sessions as our gate parameters. Beating it means higher success on the occluded tasks at matched or lower NASA Task Load Index workload, with the gap concentrated in the occluded window.
The strongest handover alternative is uncertainty-triggered traded autonomy in the style of PATO. PATO hands control to the human when a subgoal model's variance or a policy ensemble's disagreement crosses a threshold. It reports 82% precision at detecting intervention points and cut demonstration time from 109.5 s to 85.0 s. We reverse its direction, handing control to the copilot when uncertainty about the operator-plus-scene rises. We use the same copilot network and tune the threshold on the same budget as our gate.
The strongest autonomous reference is ADEPT's visuo-tactile student. It reports 8/10 on the FMB asymmetric peg with no human in the loop. This tells us whether keeping the human is worth anything on these tasks at all.
Two ablations isolate the key ingredient. One replaces the predicted visibility $\hat v_{t:t+H}$ with the current visibility $v_t$, which removes the lead time. The other removes the tactile pathway from the copilot. We found no reported result on Lacuna for any shared-autonomy method on an occluded dexterous insertion task, so every comparison on these tasks will be newly run.
Why not just…?
Why not just give DiSCo touch and a well-tuned fixed blend? This is the most serious threat. If the tactile input does all the work, a constant moderate blend may match a gated one. Our argument is that any fixed blend is wrong somewhere. High authority fights the operator during the long visible phase, which DiSCo's own need for a responsiveness parameter hints at. Low authority leaves the operator alone when they are blind. The tactile-equipped DiSCo baseline settles this directly. If it matches our method on success and workload, the gate is unnecessary and we will report that.
Why not just add an active camera so the operator can see? AV-ALOHA shows that a head-tracked camera arm helps on occluded insertion and needle threading. It also finds that stacking more camera streams sometimes hurts the learned policy. Senft et al. and the camera-control studies in the file report that managing viewpoints adds its own mental burden. An active camera also adds hardware and a new collision envelope, and it does not help when the occluder is the fingers themselves at contact. We include a condition with a second static camera chosen to see the contact zone. This tests whether cheap extra visibility closes the gap.
Why not just trigger handover on policy uncertainty? Uncertainty may well rise under occlusion. If it rises early enough, a PATO-style trigger could match ours. The difference is timing: uncertainty reacts after the observations have degraded, while geometric forecasting anticipates the loss. The predicted-versus-current visibility ablation and the uncertainty baseline together measure whether that lead time changes operator behavior. We also test a combined gate. If uncertainty alone matches, the contribution narrows to cost and simplicity, and we will say so.
Proposed Experiments
In simulation, the decisive check is sample efficiency. We measure environment steps to reach 80% success on simulated asymmetric peg insertion and dish-rack placement for three copilots trained on the same sparse reward: the gated residual copilot, the ungated residual copilot, and an ungated copilot with VLAC-style progress reward. Our prediction is that the gated copilot learns substantially faster. As a reference scale, ADEPT reports roughly 3 billion environment steps for autonomous post-training on its insertion task.
On real hardware, we will recruit a modest pool of participants, each trying every condition in counterbalanced order. They run FMB-style asymmetric insertion, a dish-rack task, and an occluded valve task. We measure success rate, completion time, collisions, NASA Task Load Index, and two behavioral load signals restricted to the occluded window: idle time and the rate of input direction reversals. We also log copilot authority over time to verify that handover begins before visibility drops.
The proposal is supported if three things hold. Gated assistance beats the tactile-equipped DiSCo and the uncertainty trigger on occluded-window success. Operator reversals and idle time drop specifically in that window. And the predicted-visibility gate beats the current-visibility gate. If the fixed-blend baseline ties on all three, the central claim is wrong.
References
- PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
- Residual Policy Learning for Shared Autonomy
- Autonomous and Teleoperation Control of a Drawing Robot Avatar
- ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation
- ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- DiSCo: Diffusion Sequence Copilots for Shared Autonomy
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- A Method For Automated Drone Viewpoints to Support Remote Robot Manipulation