Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policies

Clemence Grislain, Mathilde Kappel, Olivier Sigaud, Mohamed ChetouaniarXiv 2026 · arXiv

TL;DR

ORCHID introduces a stable online self-training framework for hierarchical robotic policies that resolves the coordination mismatch between high-level planners and low-level controllers. By filtering on-policy rollouts using environment feedback and distilling successful trajectories back into both modules via supervised learning, ORCHID allows lightweight models to match or outperform massive, heavily pre-trained Vision-Language-Action (VLA) models.

What Problem Does This Paper Address?

Decomposing complex, long-horizon robotic manipulation tasks into a hierarchy—where a high-level (HL) planner generates sparse subgoals (like visual targets) and a low-level (LL) controller executes fine-grained actions to reach them—is a powerful paradigm. However, these systems frequently suffer from the HL-LL coupling problem.

The HL planner might generate subgoals that are task-relevant but physically impossible for the LL controller to execute. Conversely, the LL controller might fail because it hasn't trained on the specific distribution of subgoals the HL planner produces. While online Reinforcement Learning (RL) could theoretically bridge this gap, applying policy gradients to diffusion-based policies is notoriously unstable due to the high-variance gradient estimates inherent in multi-step stochastic denoising. As a result, most hierarchical methods remain restricted to static, offline datasets.

Methods

The authors propose ORCHID (Online Self-TRaining for Co-adaptation in Hierarchical Diffusion policies), which adapts self-training concepts from Large Language Models (such as STaR and SPIN) to robotic manipulation.

Instead of relying on unstable gradient-based RL, ORCHID uses a simple, stable three-stage iterative loop:

  1. Policy Update: Train the HL planner (a 3D CNN U-Net diffusion model) and the LL controller (either a Diffusion Policy or an Action Chunk Transformer) independently using supervised learning on the current dataset.
  2. Rollout Collection & Filtering: Deploy the policy in the environment. For each task context, sample multiple rollouts and filter them based on a sparse, binary reward (task success). Only trajectories where the HL and LL successfully cooperate to complete the task are kept.
  3. Dataset Aggregation: Add these successful on-policy trajectories back into the training set. The authors explore two variants: ORCHID (aggregates data and retrains from scratch to prevent forgetting) and ORCHID-ft (fine-tunes the existing policy on the new data for computational efficiency).

To ensure broad exploration during rollout collection, ORCHID pairs standard environment resets with "replayed contexts"—using the terminal states of successful trajectories as new starting points to expose the model to novel object configurations.

Figure 1: ORCHID compared with common hierarchical policy training paradigms. (a): Independent training of HL and LL on a fixed offline dataset D0, prone to HL-LL coupling mismatch. (b): ORCHID iteratively refines both components through: (1) supervised updates on Dt; (2) on-policy rollout collection filtered by sparse environment reward, producing Rt — successful trajectories from which we extracts subgoals to train HL reachable by LL, and successful actions for LL conditioned on HL’s plans; and (3) dataset aggregation. Right: During rollout collection, πHL generates K subgoal sequences ˆζ for each context in C(Dt), executed by πLL via action chunks ac.

Datasets

The framework was evaluated on two primary benchmarks:

  • Franka-3Blocks: A PyBullet-based simulation environment featuring 10 language-conditioned tasks (lifting, pushing, stacking) utilizing 100 demonstrations per task (and 10 demonstrations per task for low-data regime experiments).
  • CALVIN: A challenging multi-task robotic manipulation dataset featuring 34 tasks. The authors focused on the strict D $\rightarrow$ D split under two evaluation protocols: Multi-Task Language-Conditioned (MTLC) success rate and Long-Horizon MTLC (LH-MTLC), which measures the chain of consecutive task completions (up to 5 tasks).

Key Insights and Ideas

  • Bidirectional Co-Adaptation: The core mechanism driving performance is a two-way alignment. By training on filtered on-policy successes, the HL planner learns to ground its subgoals within the actual physical capabilities of the LL controller. Simultaneously, the LL controller specializes in the exact trajectory and subgoal structures that the HL planner generates.
  • Supervised Learning as an RL Alternative: ORCHID demonstrates that filtering generative samples by a binary reward and applying standard supervised distillation can match or exceed RL performance while bypassing the optimization instabilities of diffusion policy gradients.
  • The Power of Replayed Contexts: Restricting online data collection to standard initial states limits data diversity and causes performance to collapse over long horizons. Using successful terminal states as new initial states allows the agent to robustly handle the compounding distributional shifts of long-horizon tasks.

Key Results

  • Surpassing SOTA VLAs: On the CALVIN LH-MTLC benchmark, a 4-iteration ORCHID policy achieved an average successful sequence length of 4.52, outperforming FLOWER (4.35), a 950M-parameter Vision-Language-Action model that leverages massive internet-scale pre-training and 250k robotic trajectories.
  • Outperforming Hierarchical Baselines: ORCHID consistently outperformed prominent offline hierarchical methods trained from scratch on the same data, including MDT, TaKSIE, and SuSIE, across both single-task and long-horizon metrics.
  • Robustness in Low-Data Regimes: When initialized on just 10% of the training data in Franka-3Blocks, ORCHID successfully bootstrapped a weak baseline policy (15.6% success rate) up to 72.6% after 10 iterations.
Figure 3: Both variants of ORCHID improve HD poli- cies in Franka-3Blocks and CALVIN environments. Per- formance of HD policies after 0 (offline training on D0), up to 3 iterations of ORCHID-(ft). Scatter for the mean metric and shaded areas for standard error across 3 seeds. (a) Mean SR across tasks in Franka-3Blocks. (b) Mean successful sequence length in LH-MTLC.

Limitations and Future Work

  • Computational Cost of Aggregation: Standard ORCHID requires retraining the model from scratch at each iteration to avoid catastrophic forgetting, causing training times to scale poorly as the dataset grows. While ORCHID-ft mitigates this via fine-tuning, its performance eventually plateaus.
  • The Cold-Start Problem: ORCHID relies on a non-zero initial success rate to collect its first batch of successful rollouts. If the initial policy is too weak (e.g., 0% success on highly complex tasks like stacking in low-data regimes), the self-training loop stalls. The authors suggest integrating curriculum learning strategies to address this boundary condition.

Summary updated

Abstract

Hierarchical policies decompose language-conditioned long-horizon robotic manipulation into a high-level planner and a low-level controller. However, effective coordination between HL and LL requires that both components operate on compatible subgoal distributions. We propose ORCHID, a self-training framework that enables stable online improvement of hierarchical diffusion policies by aligning planning and control through iterative refinement. By filtering policy samples via environment feedback, ORCHID identifies trajectories where the planner and controller are jointly successful and distills them back into both modules via supervised learning. This process induces a bidirectional co-adaptation: the planner grounds its subgoals in the actual reaching capabilities of the controller, while the controller specializes in the trajectory structures the planner produces. By relying on supervised distillation of filtered on-policy samples, ORCHID avoids the instability typical of online hierarchical gradient-based RL training with diffusion models. On the CALVIN benchmark, ORCHID allows a lightweight, initially weak model to outperform pure offline methods, including a Vision-Language-Action model twice its size.
BibTeX
@misc{grislain2026online,
  author = {Clemence Grislain and Mathilde Kappel and Olivier Sigaud and Mohamed Chetouani},
  title = {{Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policies}},
  year = {2026},
  eprint = {2603.05291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05291},
}
Figure