When a person teleoperates a robot arm through a camera feed, the arm itself keeps blocking the view at the worst moment: the last few centimetres before contact. Current fixes move a second camera (on a drone or a second arm) automatically. They react only after the line of sight is already lost or nearly lost. The camera then swings abruptly, and the operator must re-orient to a new viewpoint exactly when precision matters most. We propose to predict self-occlusion before it happens. The system projects the arm's moving silhouette into the camera image, wraps it in a smooth, slightly inflated envelope called a circular hull, and computes the time at which the target will cross into that envelope. That predicted time drives two things: the camera starts a slow, early repositioning, and the operator gets a short advance cue. The lead time is matched to how long people actually need to re-orient after a view change.
The bet is that, for a human in the loop, occlusion handling is limited by when the system acts, not by how fast it computes. Kinetic data structures are geometry algorithms that track when a geometric fact will stop being true and schedule a repair at that moment, instead of re-checking at every tick. They are well suited to this because their native output is a predicted failure time. If the bet is right, a forecasting camera should cut both occlusion time and operator workload relative to a reactive camera at equal compute. Benefit should also peak when the lead time roughly equals the operator's measured re-orientation latency.
Core Claim
Teleoperation research has measured that operators pay a fixed re-orientation cost whenever their context changes. PATO, a shared-autonomy system that hands control to the human only when its policy is uncertain, identifies this context switch as a main bottleneck. Its handoffs, like those of autonomous camera systems, fire at the moment of trouble, so no lead time is left to absorb that cost. We claim that self-occlusion, unlike policy uncertainty, is geometrically predictable a second or more ahead from the arm's commanded motion. Spending that prediction on early, gentle camera motion and advance operator cues should improve visibility, contact accuracy, and workload over reactive viewpoint control.
The geometric mechanism is a single certificate. Let $P(t)$ be the 2D image-plane projection of sampled points on the arm and gripper at time $t$, as seen from the current camera. Let $q(t)$ be the projected target point. The circular hull of radius $r$ is
$$H_r(P) = \bigcap \{ D : D \text{ is a disk of radius } r,\ P \subseteq D \},$$
the region bounded by radius-$r$ arcs that contains the ordinary convex hull, and that shrinks toward it as $r$ grows. The certificate "target visible with margin" fails at
$$t^* = \inf\{ t > t_0 : q(t) \in H_r(P(t)) \},$$
which we compute along a short-horizon extrapolation of the arm's motion. When the operator's command deviates from that extrapolation, we recompute.
The radius $r$ acts as a tunable safety margin, and the arc boundary gives a smooth envelope instead of a jagged silhouette. The tangent lines from $q$ to the hull tell the camera which way to orbit to keep the target in view.
Projecting into the image plane is what makes a 2D structure usable. It sidesteps the known weakness of kinetic structures in 3D, where the bounds blow up (the file cites an estimated $O(\mu^4)$ for rigid polyhedra).
Relation to Prior Work
The closest systems are two autonomous-camera teleoperation setups.
Senft et al. fly a drone around a Franka arm, using a 100 Hz local optimizer that keeps the end-effector centred and a 5 Hz global search that proposes alternative "ghost" viewpoints. They report zero collisions, 100% visibility on the e-stop and valve task, and transient occlusions of 1 to 5 seconds in more complex scenes. Their scene model is a static planar map.
Chen et al. dedicate a second Franka arm to an RGB-D camera that searches over 3D meshes to avoid the drawing arm. Compared with a manually teleoperated camera, this lowered self-reported mental and physical workload to the level of a stationary camera.
Both papers motivate this proposal: they show that autonomous cameras remove the burden of a second control loop. Both are reactive, though. Our proposal changes the timing and the operator interface, not just the optimizer.
AV-ALOHA adds a 7-DoF camera arm to a bimanual rig and learns camera motion end-to-end with imitation learning. It reaches 52% versus 30% on needle threading. This paper constrains the design. Teleoperating 21 degrees of freedom was demanding for the people collecting demonstrations, and the extra camera dimensions hurt performance on simple tasks. A geometric camera layer that needs no demonstrations avoids both problems.
The geometry literature supplies the method. Erickson et al. introduced certificate-and-event-queue tracking with hysteresis for collision detection between convex objects. We reuse the idea of an event-driven certificate with a margin, but apply it to sightlines rather than contact.
Wang gives dynamic circular hulls with $O(1)$ amortized updates and $O(\log n)$ common-tangent queries, built for the planar two-center problem. We use it as the hull and tangent machinery. We are honest that the arm silhouette has only a few hundred points, so asymptotic speed is not the point. Whether our insertion and deletion pattern fits Wang's update model is something we will check. If it does not, we fall back to recomputing the hull per event.
The standard kinetic assumption of known algebraic trajectories does not hold for human teleoperation. We treat that as the central engineering risk, and address it with short horizons and recomputation on deviation.
Research Program
We will build a geometric layer that runs on a two-arm rig in the style of Chen et al. One Franka is teleoperated. The second carries the camera on a fixed-radius orbit around the target, so camera motion lives on a small sphere.
At each event the layer has three jobs. It projects the arm's link geometry into the camera. It maintains $H_r(P)$ and the predicted failure time $t^*$. It queries tangents to choose an orbit direction that pushes $t^*$ out.
A scheduler compares $t^*$ with a lead time $\tau$. Once $t^* - t_0 < \tau$, it starts a camera move capped at a low angular velocity and shows the operator a small directional cue. The cue means "view will rotate left in about one second."
Arm motion is extrapolated from the operator's commanded velocity, using constant velocity over 0.5 to 2 seconds. A learned short-horizon predictor is an optional upgrade.
There are two analysis questions. First, how accurate and how early are the forecasts, given that human motion is erratic? Second, how does the margin $r$ trade false alarms, which cause needless camera motion and chatter, against missed occlusions?
Baselines
The naive reference point is a stationary external camera plus a wrist camera, with no viewpoint management. This is the setup that C3DM and AV-ALOHA show starves precision tasks of close-up views. A second naive reference is the manually teleoperated camera, Chen et al.'s TE-Pose condition. That paper reports it as the slowest and most demanding mode, but gives no numerical workload figures that we can find on Lacuna.
The strongest existing method is reactive autonomous viewpoint control. We will re-implement Senft et al.'s local optimizer for the camera arm, with Chen et al.'s mesh visibility term added. The local optimizer keeps the end-effector centred, holds preferred angles, and penalizes occlusion as it occurs. Its reported numbers are 100% visibility on the simple task and 1 to 5 second occlusions in complex scenes. For drawing, Chen et al. report the highest Hu-moment accuracy for their autonomous mode, but no numeric value is given in the file. For fairness, the reactive baseline gets the same camera arm, the same orbit radius, the same speed caps, the same compute budget, and the same tuning time as our method.
We count as beating it reduced total occlusion time and occlusion events during the final approach phase, combined with no worse completion time and lower NASA-TLX workload, on the same tasks and participants.
The key ablation is a brute-force forecaster. It forward-simulates the arm meshes over the same horizon at 100 Hz, ray-casts to the target, and applies distance hysteresis. It feeds exactly the same scheduler and operator cue.
Why not just…?
Why not just ray-cast the arm mesh forward in time on a GPU instead of maintaining kinetic certificates? This is the real threat. Compute is not the bottleneck, and Senft's optimizer already runs at 100 Hz. If brute-force forecasting with hysteresis matches the circular-hull forecaster on lead time and false-alarm rate, then the geometry contributes only elegance. We would report that the gain comes from forecasting and scheduling. This is why the main claim is stated about lead time, not about kinetic structures.
The geometric layer can earn its place in two specific ways. The first is a single interpretable margin $r$ with smooth arc boundaries, which may suppress the flicker of alerts and camera jitter that hysteresis heuristics produce near the silhouette edge. The second is tangent queries that directly give the orbit direction. The matched ablation settles which of these, if either, matters.
Why not just learn camera motion end-to-end, as AV-ALOHA does? Learning requires demonstrations in which people teleoperate the camera as well. AV-ALOHA reports that this is demanding, and that the extra degrees of freedom degrade simple tasks. A learned policy also gives no explicit failure time from which to schedule operator cues. As a secondary check, we will test whether head motions in DECO-50, teleoperated humanoid data with an active head, anticipate self-occlusion of the hands. If humans already lead occlusion with head motion, that both supports our premise and shows how much lead time they choose.
Why not just add more fixed cameras? Fixed views still get cut by arm links near contact, as C3DM and AV-ALOHA's static baselines (30% versus 52% on Thread Needle) show. Multiple feeds also multiply what the operator must monitor. We include a four-camera fixed array as a condition to measure this rather than assume it.
Why not just rely on touch near contact? DECO gains 20 absolute points on contact-rich tasks from tactile sensing. However, touch only helps after contact. ContactGuard notes that misaligned approaches push or disturb objects before contact detection can act. A remote human also still needs the view. Touch is complementary to this proposal, not a replacement.
Proposed Experiments
We start with an offline forecast evaluation in RLBench. Its per-camera segmentation masks give ground-truth target-occlusion events in the over-the-shoulder views. We perturb scripted demonstrations with operator-like velocity noise and measure four quantities: forecast lead time, precision and recall of occlusion events, and false-alarm rate as functions of $r$ and of the extrapolation horizon. Both the circular-hull forecaster and the brute-force forecaster are run under identical compute limits. No baseline forecasting numbers exist on Lacuna for this setup.
The main study is a within-subject user study, with participants randomized across conditions. Tasks are Senft et al.'s e-stop, breaker, and valve tasks, plus a tabletop peg-placement task where a moving cardboard obstacle is held by an experimenter. The obstacle breaks the static-scene assumption that limits Senft's method. The conditions are stationary camera, fixed four-camera array, reactive autonomous camera, brute-force forecasting with cues, and circular-hull forecasting with and without cues.
We record six measures: fraction of approach time with the target visible, number and duration of occlusions, peak camera angular velocity, contact errors and completion time following Steinfeld et al., NASA-TLX, and a re-orientation measure. The re-orientation measure is the time from the end of a camera move until the operator's end-effector command returns to a steady, accurate direction.
The decisive check is a lead-time sweep: $\tau$ = 0, 0.5, 1, and 2 seconds. We first measure each participant's re-orientation latency in a calibration block.
If the forecasting idea is right, two things should hold. Workload and contact errors should fall as $\tau$ approaches that latency, then flatten or worsen once early moves become distracting. The cue-plus-early-move condition should also beat the reactive camera on occlusion time at no cost in completion time.
The claim fails if any of three outcomes occurs. A $\tau$ of 0 performs as well as the best $\tau$, which would mean lead time does not matter. Forecasts on real teleoperation are too unreliable to keep false alarms below the reactive camera's own camera-motion rate. Or brute-force forecasting matches the circular hull on every measure, in which case the geometric component is dropped from the claim.
References
- PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
- A Method For Automated Drone Viewpoints to Support Remote Robot Manipulation
- Autonomous and Teleoperation Control of a Drawing Robot Avatar
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- Separation-Sensitive Collision Detection for Convex Objects
- On the Planar Two-Center Problem and Circular Hulls
- C3DM: Constrained-Context Conditional Diffusion Models for Imitation Learning
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- RLBench: The Robot Learning Benchmark & Learning Environment.
- Common metrics for human-robot interaction