Learning Lab Bench Skills from Human Videos as Sequences of Mechanical Events

Automated drug discovery can score millions of molecules on a computer, but only a few can be tested in a real lab, and those tests still depend on people. Liquid-handling robots do the plate pipetting. Human technicians still do the "glue" steps between instruments: screwing caps onto tubes, setting pipette volumes on click-stop knobs, and pressing buttons on mixers and thermal cyclers. General-purpose robot arms could take over these steps, but they fail at exactly the precise, contact-heavy motions involved. On a simulated biology-lab benchmark, the best current robot models screw on a centrifuge tube cap only 1.7–8.3% of the time. They also nearly always fail on multi-step sequences.

This note proposes teaching those glue skills from videos of human scientists. The key change is what the robot copies. It does not copy the human's hand path. It copies the sequence of mechanical events the human produced: thread engaged, cap seated, three detent clicks, meniscus at the line. Each event becomes a checkpoint that a robot-specific feedback controller must physically achieve before the task moves on. Any robot body can reach the same checkpoints, which makes the representation embodiment-independent. Because each checkpoint is confirmed by force, sound, or a visual state change, the robot notices when it is misaligned instead of twisting blindly. Drug discovery is the application setting and the scoring rule: success is measured as completed validation protocols per robot-hour, not as single-skill accuracy.

Core Claim

The main hypothesis concerns what should serve as the shared language between humans and robots. For contact-rich lab tasks, an event sequence transfers across embodiments better than either object motion or raw trajectories. An event sequence is an ordered list of discrete mechanical state changes, each with a detectable success condition. Human video can supply the ordering, the counts, and the tolerances of these events: how many knob clicks correspond to 200 µL, and when a cap counts as seated. It cannot supply forces. The robot therefore fills in forces with a local closed-loop policy trained in simulation for its own body.

This position builds on and differs from three lines of work.

  • Object-centric intermediate representations (the method we extend).
  • Im2Flow2Act predicts 2D object keypoint motion from human video and trains a simulation policy to reproduce it.
  • GraphMimic predicts future scene graphs of object parts and contact points.
  • Both assume that "what should happen" is visible as object motion. A cap that is cross-threaded and one that is correctly engaged look almost identical in 2D flow. A transparent pipette tip filling with liquid has no trackable keypoints.
  • Lab-automation benchmarks and digital twins (the motivation and the testbed).
  • AutoBio simulates threaded caps, detented knobs, and liquid menisci, and trains on scripted single-robot demonstrations. It attributes failures to open-loop trajectory memorization.
  • MATTERIX couples a Franka arm to an Opentrons liquid handler through hand-engineered skill primitives.
  • Neither learns from human video. Neither measures whether a skill carries over to a different robot body.
  • Kinematic retargeting (a constraint on the design).
  • DexMV shows that matching fingertip positions alone drives robot links through objects.
  • Biomechanical Comparisons Reveal Divergence of Human and Humanoid Gaits shows that copying human joint angles ignores differences between muscles and motors.
  • Both results argue against copying human trajectories when forces matter.

Research Program

We would build four things.

1. A human-scientist video corpus on AutoBio's equipment. - AutoBio digitized its equipment from real lab hardware. We would record human scientists performing its screw-cap, pipette-aspiration, thermal-mixer, and lid tasks on the same physical instruments. - Recording uses a head-mounted camera, a fixed side camera, and a contact microphone on the bench. Detent clicks and thread engagement are audible even when fingers hide them from the camera. - No robot actions are recorded.

2. An event parser. - It turns a human video into an ordered list of events from a small, shared vocabulary: contact made, thread engaged, rotation to seat, detent click, button actuated, display changed, liquid level crossed. - Each event carries a numeric parameter, such as a click count, a target fill level, or a rotation amount. - The parser is trained on AutoBio simulation rollouts, where every event is known from the physics state. It is then adapted to the real videos using a small set of hand-labeled clips. - For liquid and transparent-container events, it reads the meniscus from AutoBio's simulated liquid state rather than from keypoints. This targets the known failure of point tracking on glassware.

3. Event-conditioned local controllers for each robot body. - Each controller is trained in AutoBio's MuJoCo physics with reinforcement learning plus the existing scripted demonstrations. - Its input is the target event, proprioception, wrist force/torque, and the camera image. Its reward is reaching the event's success condition. - It is trained on a stream of randomly initialized, task-free interactions, following the Im2Flow2Act recipe of learning low-level control without task demonstrations. - We train separate controllers for AutoBio's parallel-jaw arm and for a multi-fingered hand added to the same MuJoCo scenes, the embodiment class DexMV targets. The same event sequence then drives both. - Where available, tactile pretraining uses the robot tactile streams in DECO-50.

4. A protocol layer. - It chains event-conditioned skills into the glue portion of a hit-validation workflow: uncap tube, set pipette volume, aspirate, dispense, recap, load mixer, set mixer panel. The plate work is left to a simulated liquid handler, as in MATTERIX. - The layer checks each event's success condition before moving on. On failure it retries the local segment instead of the whole trajectory, which is where AutoBio reports its steepest drops.

The drug-discovery link is deliberately narrow. Each generated candidate needs a fixed protocol, and the protocol's glue steps set the rate at which candidates can be tested. Therapeutics Data Commons shows generative models struggle under realistic limits on how many candidates can be tested ("oracle calls"). Our end metric converts skill reliability into that budget: validated compounds per robot-hour. We do not claim to fix molecular design.

Baselines

Naive reference: AutoBio's reported VLA results. A VLA (vision-language-action model, a large pretrained network that maps camera images and an instruction to robot actions) such as π0 is fine-tuned by open-loop behavior cloning on AutoBio's 100 scripted demonstrations per task.

  • AutoBio reports these success rates for current models:
  • centrifuge cap screwing: 1.7–8.3%
  • thermal mixer panel: 1.6–11.3%
  • pipette aspiration: π0 reached 42.7%; π0.5 and RDT were below 20%
  • two-task chaining: 3–15%
  • thermal cycler lid closing: π0.5 at about 96–99%
  • We rerun π0 and Diffusion Policy on the same task splits so the comparison is like-for-like rather than across papers.

Closest method from human video: Im2Flow2Act adapted to AutoBio. It is the strongest existing human-video-to-robot method with a simulation-trained low-level policy.

  • It reports an 81% average real-world success on pick, pour, drawer, and cloth tasks, and about 30% above ATM. No reported result was found on Lacuna for it on AutoBio or any threaded or detented lab task.
  • We give it the same human videos, the same simulated random-interaction data, the same compute, and the same hyperparameter search budget. The only difference from our method is that its interface is object flow rather than events.

Graph-based human-video method: GraphMimic, which reports a 33% improvement in human-to-robot transfer. It is run under the same protocol. No reported result was found on Lacuna for it on AutoBio tasks.

Closed-loop VLA with force input. This is π0 fine-tuned on AutoBio demonstrations with wrist force/torque and proprioception added to its inputs. It tests whether simply giving the model force feedback removes the gap. No reported result was found on Lacuna for this configuration.

Hand-engineered compliance controller. This is a scripted impedance controller for screwing and knob turning, a controller that deliberately yields to contact forces, with per-task hand-tuned thresholds. It is the classical engineering answer. No reported result was found on Lacuna for it on AutoBio.

Ablations isolating the key ingredient: 1. Our system with events given by the simulator (oracle events), to separate parsing error from control error. 2. Our system with the video removed, so event lists come from a hand-written protocol template. 3. Our system without the audio channel.

What counts as beating the field: - Cap screwing and thermal-mixer success above 50%, which is several times the AutoBio range. - Three-task chaining above 50%. - Both reached on the multi-fingered hand from the same videos, with at most a 15-point drop relative to the parallel gripper, beating Im2Flow2Act's cross-body drop.

Why not just…?

Why not just use a dedicated liquid handler? For plate pipetting, that is the right answer, and we keep one in the loop. MATTERIX itself needed a Franka arm alongside its Opentrons to move glassware and operate instruments. The steps AutoBio models are threaded tubes, detented knobs, and clamping lids built for human hands. A gantry cannot perform them without re-engineering the instruments. Our throughput metric counts only these glue steps, so the comparison does not overclaim.

Why not just collect more robot demonstrations and fine-tune a VLA? AutoBio already gives 100 demonstrations per task and still reports single-digit cap-screwing success. Its own diagnosis is that the failure comes from open-loop memorization, not data volume. To test this directly, we run a data-scaling curve: π0 at 100, 300, and 1,000 simulated demonstrations. If success climbs toward our numbers, the event interface is unnecessary for single-body performance. The remaining claim would then rest on cross-body transfer, which needs new robot data for every body under that approach.

Why not just write a force-feedback controller? This is the most serious threat. A tuned impedance controller may well screw a cap reliably on one robot. What it cannot do on its own is read, from a video, how many clicks this protocol needs, whether this scientist seats the cap to a stop or a quarter-turn short, and in what order steps occur. It also must be retuned by hand for each body. We run the scripted controller as a baseline. If it matches our single-skill success, the contribution narrows to specifying events from video and to per-body learning in place of hand-tuning. The event-from-template ablation measures exactly how much the video adds.

Proposed Experiments

Experiment 1: single-skill success on AutoBio's precision tasks. - Tasks: cap screwing, pipette aspiration, thermal mixer panel, and lid closing as a sanity check. - Measures: success rate over 300 randomized episodes per task, for all baselines, on the parallel gripper.

Experiment 2: cross-embodiment transfer. The same human videos drive the multi-fingered hand. We compare the drop from gripper to hand across our method, Im2Flow2Act, and GraphMimic. This is the direct test of the core claim.

Experiment 3: protocol chaining. The full glue sequence of a compound-validation workflow is chained with the simulated liquid handler. We report end-to-end completion rate and simulated compounds processed per robot-hour. Failures are attributed to individual events using the ablations above.

Decisive check 1: flow versus events. The critical result is Experiment 1 on cap screwing and aspiration comparing oracle-event, parsed-event, and flow interfaces. If flow matches events once both have force input, the claim that events are the better cross-body language is false and we will report it as such.

Decisive check 2: does video add anything? Compare the parsed-from-video event lists against the template-derived lists on protocols where scientists deviate from the written procedure. If they perform the same, the human video is not contributing and the project reduces to a force-aware skill library.