Robots that learn from demonstrations face two opposite pressures when they work beside a person. To keep small mistakes from piling up, the robot should commit to short open-loop plans: predict a "chunk" of several future actions at once and execute it without re-deciding every step. To stay useful to a partner who changes speed, posture, or strategy partway through a task, the robot should re-decide often. Practitioners usually reconcile the two by predicting overlapping chunks and averaging them over time. We point out that this averaging is a repeated low-pass filter, the same operation that makes deep graph networks lose detail. It causes a specific failure: the robot's response to a sudden change by its human partner gets smoothed away, arriving late and diluted.
Our proposal borrows the fix that graph networks use, which is to keep a fixed share of fresh, unsmoothed signal at every step. We add one change that matters for robots. The share of fresh signal is not constant. It rises only when the human partner does something the robot did not predict, and it stays low when the error comes from the robot's own drift. The claim is that the source of the deviation should decide whether the robot re-plans. Deviations caused by the robot's own actions are best absorbed by committing to the current chunk. Deviations caused by the partner adapting should break the commitment.
Core Claim
Let $\hat a_t^{(s)}$ be the action for time $t$ predicted by the chunk generated at time $s \le t$, for chunk length $K$. Standard temporal blending executes a weighted average $\bar a_t = \sum_{k=0}^{K-1} w_k \hat a_t^{(t-k)}$ with fixed weights $w_k$. If a human strategy change first becomes visible at time $t$, only the newest chunk reflects it. Under uniform weights, its influence is $1/K$ at first and takes about $K$ steps to dominate. This lag follows directly from averaging $K$ predictions, of which only the fresh ones carry the new information. Repeating the blend step after step is a temporal analogue of Laplacian smoothing, the neighbor-averaging inside graph networks: high-frequency changes are exactly what disappears.
Simple and Deep Graph Convolutional Networks (GCNII) showed that ordinary skip connections only delay this collapse in deep graph networks. What prevents it is an initial residual: re-injecting a fixed fraction $\alpha$ of the raw, unsmoothed input at every layer. That paper supplies our method. We execute
$$a_t = (1-\alpha_t)\,\bar a_t + \alpha_t\,\hat a_t^{(t)}, \qquad \alpha_t = \sigma\big(\gamma\,(e^{H}_t - \tau)\big),$$
where $\hat a_t^{(t)}$ is the newest chunk's action, $e^H_t$ is the error between the human motion the policy predicted when the current chunks were generated and the motion actually observed, $\sigma$ is the logistic function, and $\gamma, \tau$ are a learned sharpness and threshold. A fixed $\alpha$ is the direct GCNII analogue. The gating on partner surprise, rather than on robot tracking error, is what is new.
The reason for the gating comes from Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces. That paper proves chunking turns exponential error compounding into linear or horizon-free error when dynamics are open-loop stable. It warns that the guarantee fails under fast external disturbances. A human partner who adapts is such a disturbance, but only part of the time. We therefore predict that a fixed $\alpha$ sits on a trade-off curve between drift and response lag, and that partner-gated $\alpha$ moves past that curve.
Relation to Prior Work
Three works motivate the problem. Co-GAIL trains human and robot policies jointly and infers a latent strategy code from the human's early motion. It acts step by step and never confronts the chunking trade-off. We reuse its partner policy $\pi_H$ as a simulated partner and its benchmarks. Adaptive Human-Robot Collaboration using Type-Based IRL tracks a drifting human "type", such as fatigue, with a recurrent belief and switches robot behavior accordingly. It works in discrete sorting tasks, whereas we address how fast a continuous controller can act on such a belief. Fast Motion Prediction for Collaborative Robotics reports 70% goal-prediction accuracy within the first 400 ms of a reach. This tells us partner changes become detectable well inside a typical chunk window, so lag from averaging is a real cost and not a theoretical one.
Two works constrain the design. The action-chunking paper above rules out simply re-planning every step, since that restores compounding error. Rethinking Safe Control in the Presence of Self-Seeking Humans shows that humans exploit a predictable, conservative robot. Excessive smoothing is a form of predictability, so lag may change human behavior and not merely delay the robot.
We deliberately do not build a deep graph network over human and robot skeletons, even though the research file flags that as a gap. A flat or transformer policy would likely match it, and that work would test architecture rather than interaction. Over-smoothing enters as a mechanism and a fix applied along the time axis. Second-order damped updates in the style of DuSEGO are a possible alternative blending rule. We treat them as a secondary variant because of their added compute and latency.
Research Program
We will build a chunked visuomotor imitation policy with an auxiliary head that predicts the partner's motion over the chunk horizon. The surprise signal $e^H_t$ comes from that head. The gate parameters are fit by behavior cloning on demonstrations augmented with partner-response perturbations. During data collection, the partner policy switches strategy at random times, and the clean demonstrator action after the switch is logged. This mirrors the noise injection of the chunking paper, but the perturbation is applied to the partner's strategy rather than as isotropic action noise, which targets the human-induced shift that Gap 3 of the file identifies.
We will also measure response lag analytically and empirically as a function of $K$ and $\alpha$. This shows how much of the failure of chunked policies around humans is explained by the smoothing view.
Baselines
The naive reference is per-step behavior cloning with no chunking. On HR-Handover with real operators, Co-GAIL reports BC and GAIL below 25% success, which bounds what we expect.
The strongest existing method is Co-GAIL itself, which reports 70–75% success with real human operators on HR-Handover. It also reports 53.3% replay success on 2D-Fetch-Quest versus 30% for MA-GAIL. Because 2D-Fetch-Quest is a grid world where chunking is not meaningful, we use it only as a check on partner modeling. We will run Co-GAIL from the same RoboTurk-collected demonstrations, with the same backbone width, matched gradient steps, and the same hyperparameter search budget as our method. Beating it means at least matching its success rate while reducing the time from a partner strategy switch to a correct robot response.
The chunked baselines are uniform temporal blending, exponential blending with tuned decay, and fixed-$\alpha$ initial-residual blending (the GCNII analogue). We sweep chunk length $K$ and $\alpha$ over a grid at equal cost and report the full trade-off curve, not a single point.
The key ablation replaces partner surprise $e^H_t$ with robot tracking error as the gate input. If that variant performs as well, our claim about deviation source is wrong.
Why not just…?
Why not just shorten the chunks? Shorter chunks reduce lag but bring back compounding drift, the trade-off the chunking paper formalizes. Shortening is equivalent to moving along the fixed-$\alpha$ curve. Our K-sweep reports exactly this curve. If some fixed $K$ matches gated blending on both lag and drift, we will report that the gating adds nothing.
Why not just add a reactive compliance layer under the chunked policy? Impedance control limits contact forces but does not re-plan intent. Anticipatory Control for Proactive Human Robot Collaboration finds that purely reactive compliance shifts physical effort onto the human. We include compliance in all variants, so the comparison isolates planning.
Why not just infer the partner's strategy, as Co-GAIL does, and condition on it? Strategy inference tells the robot what changed. It does not stop old chunks from diluting the new plan during execution. We test a Co-GAIL-style latent-conditioned chunked policy with and without gating. If conditioning alone closes the lag, the threat is real and the gate is unnecessary.
Why not just use DAgger-style human corrections? Continuous online labeling during physical interaction is costly, and ThriftyDAgger exists precisely to ration such queries. We report a budget-matched ThriftyDAgger run in simulation only.
Proposed Experiments
HR-Handover and HR-Handover with switches. On HR-Handover we measure success and the latency from a scripted change in handover location to the moment the robot commits to the new target. We use simulated Co-GAIL partners first, then a small real-operator study. A switched variant changes the partner's strategy mid-episode.
HR-SeqManip. We evaluate on HR-SeqManip, for which no reported result was found on Lacuna, so all numbers will be new.
robomimic tool_hang with partner disturbance. For a cleaner test of open-loop stability, we add scripted "partner" disturbances to robomimic tool_hang. These are object nudges that are visible to the partner-prediction head, alongside matched robot-only noise. The chunking paper reports that success rises with chunk length there, but gives the values only in a figure. We will therefore reproduce its no-disturbance curve as a sanity check before adding disturbances.
Dressing with arm motion. In Softgym with SMPL-X bodies, we evaluate dressing with scripted arm motion and report dressed-arm ratio. For context, One Policy to Dress Them All reports 86% of total arm length dressed in real trials. We found no reported result on Lacuna for dressing with mid-trial arm motion.
Decisive checks. The proposal succeeds if three things hold. First, gated blending dominates the fixed-$\alpha$ and $K$ trade-off curves on the partner-switch tasks. Second, gating on robot tracking error does not reproduce the gain. Third, under robot-only disturbances, gated blending matches long-chunk performance, because the gate should stay closed when the human has not changed.
References
- Simple and Deep Graph Convolutional Networks
- Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces
- Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
- Adaptive Human-Robot Collaboration using Type-Based IRL
- Fast Motion Prediction for Collaborative Robotics
- Rethinking Safe Control in the Presence of Self-Seeking Humans
- DuSEGO: Dual Second-Order Equivariant Graph Ordinary Differential Equation
- An experience-driven robotic assistant acquiring human knowledge to improve haptic cooperation
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments