Robot imitation learning trains a policy to copy pairs of what the robot saw and what it did next. Those pairs are only correct if the camera frame, the joint reading, and the touch reading were stamped with the same instant. Most demonstration pipelines stamp each stream in software, when the operating system gets around to it. That leaves offsets and jitter of roughly 10 to 50 ms, so a frame is often paired with an action from a slightly different moment. Hardware synchronization fixes this by sending electrical trigger pulses from one clock to every sensor, which brings the error down to microseconds. It costs wiring, custom boards, and engineering time. Data-scarce robot learning has so far chosen scale over timing: ROBOTURK uses phones over WebRTC, TeleMoMa streams commands over Wi-Fi, and Kaiwu aligns many devices with software clocks.
We propose to measure the trade directly and express timing error in units of demonstrations lost. The central experiment logs the same physical demonstrations twice, once with hardware-triggered timestamps and once with ordinary software timestamps. The two resulting datasets differ only in timing. We then ask how many extra software-timed demonstrations a policy needs to match one trained on hardware-timed ones, and which part of the timing error causes the loss: constant offset, random jitter, or phase skew between streams. The answer tells a lab with a fixed budget whether to spend on more demonstrations or on a trigger line.
Core Claim
Timing error in demonstrations is a form of action-label noise. Its cost scales with a single dimensionless ratio,
$$r = \frac{v\,\sigma_t}{\varepsilon},$$
where $v$ is the typical end-effector or object speed during the task, $\sigma_t$ is the standard deviation of the observation–action timing error, and $\varepsilon$ is the spatial tolerance the task demands. The form follows from the fact that a timing error of $\sigma_t$ shifts the paired state by about $v\sigma_t$. That shift only matters relative to how precisely the task must be executed.
We make three predictions. First, the extra demonstrations needed rise sharply once $r$ approaches order one, and stay negligible below that. Second, the constant part of the timing error can be removed after the fact in software, so it should cost almost nothing once corrected. Third, the time-varying jitter and the skew between streams cannot be recovered from the data, and these are where hardware synchronization earns its cost.
If all three hold, we can tell a lab which tasks need hardware synchronization: fast, tight-tolerance, contact-rich ones. For slow tabletop picking, cheap Wi-Fi collection is fine.
Relation to Prior Work
The closest work handles a different delay. Delay-Aware Diffusion Policy conditions a diffusion policy on the inference delay between observation and execution, and assumes the recorded sensor streams themselves are perfectly aligned. We study misalignment baked into the training data, which delay conditioning at deployment does not touch. AsyncBEV warps driving-perception features using a known sensor offset $\Delta t$. It motivates software re-timing as a serious competitor, but it targets 3D detection rather than action labels and requires $\Delta t$ to be known. AsyncVLA uses the word "asynchronous" for the order in which action tokens are generated and assumes synchronized inputs. ORCA aligns demonstration pacing at the level of subgoals, which is far coarser than the millisecond errors at issue here.
On the hardware side, two papers supply the method. Liu et al. synchronize seven tactile sensors on a gripper to within 30 µs over an I2C/SPI hub. Wan et al. trigger four cameras and an IMU from an FPGA, a reprogrammable logic chip. We reuse their style of trigger design but never train a policy on top of it in the way they do not.
Three papers motivate the gap by showing how data is collected in practice. M2DGR reports software synchronization within about 10 ms. TeleMoMa and ROBOTURK scale collection over networks with variable latency. Large aggregates such as RDT-1B inherit whatever timing their sources had.
Research Program
We would build a dual-logged demonstration rig. It consists of a single arm with joint encoders, two global-shutter cameras, and, in a second phase, fingertip tactile sensors on a bus modeled on Liu et al. A microcontroller sends trigger pulses and records hardware timestamps. In parallel, the standard robot software stack records its own receive-time stamps for every message. Each demonstration therefore yields two aligned copies whose difference is exactly the real-world software timing error. Recording that difference also gives us the empirical distribution of offset, jitter, and skew for typical setups, which no paper in the file reports for manipulation.
We would then decompose the measured error into three components: a constant per-stream offset, slow drift, and residual jitter. We build hybrid datasets that contain only one component at a time. This isolates which component costs data.
Next we would fit a software re-timing estimator. It aligns joint velocity with image motion to recover offset and drift per episode, using no hardware. We validate it against the hardware ground truth. This estimator is the realistic fix a lab without trigger hardware would apply, and the program is designed so it might win.
Finally, simulation extends the sweep beyond what the rig can reach. We inject the measured error distributions, scaled across $r$, into Robomimic, Push-T, and ManiSkill dynamic tasks. Simulation gives perfect ground-truth timing and lets us vary $v$ and $\varepsilon$ independently.
Baselines
The naive reference trains on software-timestamped demonstrations exactly as logged. This is how most existing datasets are used, and it sets the floor.
The upper reference trains on hardware-timestamped copies of the same demonstrations, with the same policy, the same number of demonstrations, the same training steps, and the same hyperparameter search budget.
The strongest software competitor is software re-timing: offset and drift estimated by cross-correlating joint velocity with optical flow, then applied before training. It is the strongest because it is free and attacks the error component we expect to dominate. It counts as beaten only if hardware-timed data outperforms it at matched demonstration counts by more than the seed-to-seed spread, on tasks with $r$ near or above one.
A second competitor is time-shift augmentation: training on randomly shifted observation–action pairs so the policy learns to tolerate misalignment. It receives the same tuning budget.
A third competitor applies Delay-Aware Diffusion Policy-style conditioning on the estimated offset. The strongest reported number in this family is that DA-DP holds 96% success on a ManiSkill rolling-ball task under 0.05 s delay, where vanilla Diffusion Policy falls to 20% (Delay-Aware Diffusion Policy). That delay is inference delay, not demonstration timing, so we treat it as a reference point rather than a number to beat directly.
The underlying policy is Diffusion Policy, which reports 95% real-world Push-T success. For the tactile phase we use a force-output variant following TacDiffusion, using its released code TacDiffusion codebase.
The key ablation is "hardware offset only." It applies the hardware-measured constant offset but keeps the software jitter. This isolates whether hardware synchronization helps beyond what a perfect offset estimate would give.
Why not just…?
Why not just collect more demonstrations? This is the question the study answers, not a competing method. The headline output is the exchange rate: how many software-timed demonstrations equal one hardware-timed demonstration at a given $r$. If that rate stays near one across tasks, the honest conclusion is that collecting more data wins, and we will report it.
Why not just estimate the offset in software? This is the real threat. For slow tasks dominated by constant offset, we expect it to close most of the gap. The dual-logged rig settles the question by measuring the estimator's residual error against hardware truth. The "hardware offset only" ablation then shows whether the remaining jitter still costs demonstrations.
Why not rely on action chunking or augmentation to absorb the jitter? Predicting chunks of future actions averages over some noise. However, the DA-DP results show vanilla Diffusion Policy is not robust to 50 ms of mismatch in dynamic tasks, and TacDiffusion reports that removing its high-rate filter drops success by more than 9% on tight-clearance insertion. Both are suggestive rather than direct evidence about demonstration timing, so augmentation runs as a tuned baseline instead of being argued away.
Why not train on passive human video and skip sensors altogether? Video carries no contact force, and the tasks where we predict timing matters most are the contact-rich, tight-tolerance ones, which video-only methods do not address. This alternative is orthogonal to the question rather than a competitor.
Proposed Experiments
On the real rig we would collect about 200 demonstrations each of three tasks spanning $r$. The slow task is pick-and-place. The intermediate task is pushing a moving object, in the style of Push-T. The fast, tight-tolerance task is peg insertion with tactile sensing. For each task we train policies at 25, 50, 100, and 200 demonstrations under every baseline, with five seeds, and evaluate 50 rollouts per policy. The metric is success rate. From the resulting curves we read off the exchange rate between software-timed and hardware-timed demonstrations.
In simulation we sweep $\sigma_t$ from 0 to 50 ms using the measured error distributions on Robomimic, Push-T, and ManiSkill rolling-ball tasks, and plot success against $r$.
The decisive check has two parts. First, the curves from all tasks should collapse onto one function of $r$. Second, for tasks with $r \ge 1$, the hardware-timed data should require substantially fewer demonstrations than software re-timing does.
If the curves do not collapse, the ratio is the wrong summary, and we report which task property breaks it. If software re-timing matches hardware everywhere, the conclusion is that data-scarce labs should not buy synchronization hardware. That is a useful negative result for the field.
References
- Delay-Aware Diffusion Policy: Bridging the Observation-Execution Gap in Dynamic Tasks
- AsyncBEV: Cross-modal flow alignment in Asynchronous 3D Object Detection
- AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Models
- Imitation Learning from a Single Temporally Misaligned Video
- Large-scale Deployment of Vision-based Tactile Sensors on Multi-fingered Grippers
- An Energy-Efficient Quad-Camera Visual System for Autonomous Machines on FPGA Platform
- M2DGR: A Multi-Sensor and Multi-Scenario SLAM Dataset for Ground Robots
- TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation
- ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- TacDiffusion: Force-Domain Diffusion Policy for Precise Tactile Manipulation