When a two-armed robot does the delicate part of a task, such as threading a part, folding a sleeve, or twisting a lid, both arms converge on the object and cover it from the cameras. The policy loses sight of the object at the moment it matters most. Tools for reconstructing moving scenes cannot fix this, because they treat the robot's arms as unknown moving things. They either delete the arms as noise or model them as free-floating objects. The robot, however, knows exactly where its own arms are: joint encoders and the robot's geometric model place every link to within millimeters. It also knows when a gripper has closed on the object. From that point on, whatever is held moves with the gripper. In a bimanual grasp the object is held from two sides, so the two known gripper motions pin down how the object moves even when no camera can see it.
We propose a streaming dynamic reconstruction that builds the robot's own arms from their known kinematics instead of inferring them. It uses the grasp events of both grippers as motion anchors for the shared object. The reconstruction runs as a slower background process. A fast bimanual policy reads from it: an up-to-date 3D model of the workspace with the arms removed, plus an estimate of the object's pose. The core claim is that most of the "terminal blind spot" in bimanual manipulation can be recovered from information the robot already has. Masking the arms is not enough, because the object keeps moving while it is hidden. What recovers it is carrying the object forward through the occlusion using the kinematics of both grippers.
What is already done, and what is not
The research file contains several close neighbours, and each leaves the key gap open.
BIGS reconstructs two human hands with rigid objects from a single video. It uses 3D Gaussian Splatting, a representation that models a scene as a cloud of small colored 3D blobs rendered by projecting them onto the image. It relies on a parametric human-hand model and on diffusion-based hallucination to fill in unseen geometry. It runs offline, has no robot, and fills occluded regions with a generative prior. We fill them with kinematic constraints instead.
HOGSA anchors Gaussians to hand and object meshes to synthesize training data. The idea of attaching Gaussians to known geometry is the one we reuse for robot links, but HOGSA does not track anything online.
Robot See Robot Do tracks articulated object parts in 4D and executes the motion on a bimanual YuMi. It needs a pre-scan, runs offline at about 1.4 s per frame, and tracks the human demonstration rather than the robot's own execution.
ManiGaussian learns an action-conditioned deformation field over Gaussians as a world model. It is single-arm, and its deformation is learned rather than taken from joint telemetry.
GeoPredict uses 3D Gaussian prediction only as a training-time auxiliary and discards it at inference.
Flex-π predicts 2.5D pointmaps alongside bimanual actions but keeps no persistent 3D state across occlusion.
On the reconstruction side, Split4D and RobustSplat supply the relevant machinery. Split4D decomposes a 4D scene without 2D trackers, and RobustSplat delays densification, the step where Gaussians are split or cloned, so that transient objects do not leave floating artifacts. Neither method uses any knowledge of the moving bodies. Split4D explicitly reports boundary blurring under tight contact, which motivates us to supply the boundaries from kinematics.
No existing work does both things at once: treat the manipulators as known rigid bodies driven by telemetry, and use dual-gripper grasp constraints to propagate an occluded object's state online.
Core Claim
The representation has three parts. Robot-link Gaussians are rigidly attached to each link and posed by forward kinematics, meaning the computation of link poses from joint angles. These Gaussians are rendered but never optimized for position, so any photometric error they cause can be attributed to them. Background Gaussians are static and updated with densification only in pixels not covered by the rendered robot mask. Object Gaussians carry a pose, or a low-dimensional deformation.
When gripper $g$ is closed on the object, the object's motion is tied to the motion of that gripper. Let $T_g(t)$ be the gripper's pose from kinematics at time $t$, and let $t_g$ be the moment of grasp. For a rigid object held by both grippers, the predicted object pose from gripper $g$ is $T_g(t)\,T_g(t_g)^{-1}\,X(t_g)$, where $X$ is the object pose. We fuse this with whatever visual evidence remains through the objective
$$ \min_{X(t)} \; \mathcal{L}_{\text{photo}}\big(X(t);\, M_{\text{vis}}(t)\big) + \lambda \sum_{g \in \{L,R\}} c_g(t)\, d\big(X(t),\, T_g(t)\,T_g(t_g)^{-1}X(t_g)\big). $$
Here $\mathcal{L}_{\text{photo}}$ is the rendering loss restricted to the unoccluded pixels $M_{\text{vis}}(t)$, and $d$ is a distance between poses. The weight $c_g(t)\in[0,1]$ expresses confidence that gripper $g$ is holding firmly, taken from gripper width and effort. The constant $\lambda$ balances the two terms.
The two anchor terms play a bimanual-specific role. When both grippers hold a rigid object, their predictions must agree. Disagreement between them is a signal that the object has slipped, and it lowers the corresponding $c_g$. For deformable objects such as cloth, the two grasp points become boundary conditions on a coarse deformation graph, which is fitted to the visible portion.
Research Program
We build the system in three stages.
Offline reconstruction. First we build it on synchronized multi-view recordings and measure how well it recovers object pose during occlusion. We use simulated bimanual tasks from the RoboTwin suite used by Flex-π, where ground-truth object poses exist.
Streaming at control rates. Second, we make the reconstruction run as a streaming process at a rate decoupled from control. It updates background and object state at a few hertz. It publishes two outputs: the object pose estimate, and a rendered view of the scene with the arms removed from a fixed virtual camera. This follows the virtual-camera idea of RVT, extended to a dynamic scene.
Policy integration. Third, we attach these outputs to a bimanual policy as extra inputs. We use a TwinVLA-style twin-arm model, or a diffusion policy. The policy runs at its native rate and reads the latest published state, so reconstruction latency never blocks action generation.
We also run a transfer check with human data. On ARCTIC we substitute the human-hand model for the robot's kinematic model. This tests whether contact anchoring still helps when the kinematics themselves come from estimation rather than encoders.
Baselines
The naive reference is the bimanual policy trained on the raw camera streams alone, with no reconstruction. It shows how much of the terminal blind spot matters at all. It is trained with identical demonstrations and an identical tuning budget.
The strongest simple competitor is kinematic masking. We render the robot's silhouette from its geometric model and joint angles, exclude those pixels, and run a RobustSplat-style reconstruction of whatever remains visible. This isolates whether our gain comes merely from knowing where the arms are. It is the most likely baseline to be good enough, so it gets the same camera setup and update rate as our method.
The strongest robot-agnostic dynamic reconstruction is Split4D. Its reported mIoU of 0.893 against 0.592 for tracker-based SA4D on the Multi-Human dataset shows it is the best available decomposition without trackers. We run it on the same multi-view streams.
For articulated-object tracking we compare against the 4D differentiable part models of Robot See Robot Do, which reports 7.5 mm average tracking error and 60% end-to-end success on the YuMi. Beating it means lower tracking error during frames where the parts are occluded by the robot, at comparable or lower per-frame cost.
For the end-to-end policy comparison we use a Flex-π-style world-action model, which predicts pointmaps and has no persistent 3D state. Flex-π reports 2–6× over π0.5 on its real YAM tasks and 76% versus 33% on self-repair. For RoboTwin specifically, there is no reported result found on Lacuna for a numerical success rate. On real hardware the reference number is TwinVLA's report of 71.0% for TwinVLA against 76.0% for π0 on the Anubis robot.
Two ablations isolate the key ingredient: removing the grasp-anchor term, which leaves masking plus persistence, and removing the cross-gripper consistency check, which uses a single anchor. Beating kinematic masking by a clear margin on occluded-frame object pose error, and turning that into higher task success on occlusion-heavy tasks, is the result that would justify the method.
Why not just…?
Why not just mask the arms using the robot model and reconstruct the rest? This is the real threat, and it is our main baseline. Masking removes arm artifacts but provides no information about an object that moves while hidden. After a two-handed grasp, the object may be almost entirely covered for the whole manipulation phase. If occluded-frame pose error for masking stays close to that of our full method, the grasp-anchor term is not earning its place. In that case we would report masking as the recommended practical solution.
Why not just give the policy image history and let it learn memory end-to-end? Monolithic scaling is expensive for bimanual robots. TwinVLA documents RDT-1B using about 2,400 hours of data and reaching only 45.0% real-world success. Any3D-VLA finds that implicit depth channels underperform explicit 3D representations. Neither finding directly tests memory under self-occlusion, though. Our Flex-π-style and raw-history baselines on occlusion-stratified tasks settle this directly.
Why not just add wrist cameras or tactile sensors? Wrist cameras are themselves occluded when the grippers approach contact, as noted in the self-occlusion direction (Overcoming Visual Self Occlusion). DECO shows tactile input helps contact-rich tasks, raising success from 72.25% to 82.50%, but touch is local and does not give object pose. We include a wrist-camera variant of the naive baseline. The approaches are complementary, and we expect gains to persist on top of wrist views.
Why not just use 2D video segmentation to remove the arms? Split4D shows that tracker-dependent pipelines degrade badly under tight interaction. A mask from the robot model is strictly more reliable than a learned one for the robot's own body, so this alternative is dominated by our masking baseline.
Proposed Experiments
Simulation. In RoboTwin we select bimanual tasks with a rigid shared object, an articulated object, and a deformable object. We stratify every evaluation frame by the fraction of the object hidden by the arms. The primary metric is object pose error, or Chamfer distance for deformables, as a function of occlusion fraction. The decisive check is that the full method's error curve stays flat as occlusion rises past half the object, while the masking baseline's error curve climbs.
We then inject gripper slip in simulation to test the cross-gripper consistency signal. Success here means detecting slip before pose error exceeds a set tolerance, where the single-anchor ablation silently drifts.
Downstream policies. We train identical policies with and without the published reconstruction outputs. We report task success overall and on the occlusion-heavy subset. Latency is also measured: the policy's action rate must be unchanged.
Real robot. On a real bimanual arm we replicate a subset of the articulated tasks from Robot See Robot Do and compare against its reported 7.5 mm tracking error.
Human-hand transfer. On ARCTIC we report Chamfer distance and hand error against the relative gains reported by BIGS. BIGS reports a roughly 40% Chamfer reduction over HOLD, and PSNR rising from about 12.8 to 24.8. This tests whether contact anchoring still helps with estimated rather than measured kinematics. A negative result there would mark the method as specific to robots with encoders, which is still its intended use.
References
- BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation
- Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction
- ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation
- GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation
- Flex-$π$: A Multi-Stream World-Action Model with Compute Flexibility
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- RVT: Robotic View Transformer for 3D Object Manipulation
- TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models
- Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter