Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division

Dafni Antotsiou, Carlo Ciliberto, Tae-Kyun Kim

Abstract

Deep imitation learning (IL) typically requires a vast number of expert demonstrations, a requirement that scales poorly as the number of tasks increases. While multi-task learning (MTL) aims to alleviate this by sharing knowledge across tasks, it often suffers from "negative transfer," where task-specific features interfere with one another. We introduce Modular Adaptive Policy Selection (MAPS), a framework that performs multi-task imitation by decomposing complex behaviors into shared, primitive "proto-policies." These modules are adaptively combined by a selector mechanism trained to balance task-specific requirements with cross-task commonalities. Our experiments across diverse robotic benchmarks demonstrate that MAPS outperforms single-agent baselines, task-conditioned networks, and state-of-the-art meta-learning methods like MAML, while autonomously discovering meaningful sub-behaviors.

Introduction

Imitation learning allows robots to acquire complex skills by mimicking human experts. However, the data-hungry nature of deep learning means that mastering a suite of diverse tasks—such as opening drawers, pushing buttons, and picking up objects—usually requires a prohibitive amount of demonstration data.

Multi-task learning offers a theoretical solution: if tasks share underlying physical principles, a model should be able to learn them more efficiently together than in isolation. In practice, however, standard MTL architectures often fall victim to negative transfer. When a single neural network is forced to represent conflicting motor skills (e.g., the precise wrist rotation for a screwdriver vs. the linear push for a button), the resulting interference can lead to performance that is worse than training separate models for each task. MAPS addresses this by moving away from monolithic architectures in favor of a modular, adaptive approach.

The MAPS Framework

MAPS is built on the premise that complex tasks are composed of simpler, reusable primitives. The architecture consists of two primary components: a library of proto-policies and an adaptive selector.

1. Proto-Policies (Modules)

Instead of one large policy, MAPS maintains a set of parallel neural network modules. Crucially, these modules are "blind" to the specific task ID; they only receive the environment state. This constraint forces the modules to learn general-purpose physical maneuvers—such as reaching toward an object or grasping—that are applicable across multiple scenarios.

2. The Adaptive Selector (G)

The selector acts as the coordinator. It observes both the environment state and the task ID to determine which proto-policies should be active at any given moment. By outputting a set of weights, the selector creates a weighted combination of module outputs to produce the final action. This allows the system to transition between different behaviors (e.g., moving to a handle, then pulling) seamlessly.

3. Optimization and Regularization

To ensure the modules learn distinct and useful behaviors, the authors employ a composite loss function with four specific regularizers:

  • Sharing: Encourages different tasks to utilize the same modules for similar states, promoting positive transfer.
  • Exploration: Prevents the system from collapsing into a single "super-module," ensuring the entire library is utilized.
  • Sparsity: Forces the selector to choose only a few modules at a time, which helps in identifying distinct physical primitives.
  • Smoothness: Penalizes abrupt changes in module selection to ensure the robot's movements remain fluid and stable.

Experimental Evaluation

The researchers evaluated MAPS against three distinct types of variation to test its robustness:

  1. Environmental Variation (HalfCheetah Gravity): Five tasks where the robot must run under varying gravity levels.
  2. Model Variation (HalfCheetah Modified): Eleven tasks where the robot's physical proportions (thigh, foot, etc.) are altered.
  3. Task Variation (MetaWorld MT10): A challenging benchmark involving ten diverse manipulation tasks using a Sawyer robot arm, including reaching, pushing, and pick-and-place operations.

Key Insights and Results

Superior Performance and Positive Transfer

MAPS consistently outperformed state-of-the-art baselines, including MAML (Model-Agnostic Meta-Learning) and standard task-conditioned networks. In the MT10 benchmark, MAPS achieved success rates over 10% higher than the next best method. Furthermore, MAPS outperformed single-agent baselines in over 90% of the tested scenarios, indicating that the framework successfully harvested knowledge across tasks rather than just avoiding interference.

Autonomous Task Decomposition

One of the most significant findings was the system's ability to discover sub-behaviors without human supervision. For example, in manipulation tasks, the selector learned to activate specific modules for "approaching" and others for "interacting." When tasks shared similar phases—like grasping a handle—the selector automatically reused the same proto-policies, effectively discovering the "intrinsic dimensionality" of the task space.

Efficiency through Modularity

The modular approach proved highly efficient. In the MT10 suite, a MAPS configuration with only five modules performed comparably to ten separate, specialized agents. This suggests that complex robotic task sets can be compressed into a small library of fundamental skills.

Conclusion and Future Work

MAPS provides a scalable path for multi-task imitation learning by balancing the benefits of shared representations with the necessity of task-specific expertise. While the framework currently requires careful tuning of the regularization hyperparameters ($\lambda$), it demonstrates a clear advantage in handling diverse task sets without the performance degradation typical of standard MTL.

Future research directions include using the pre-trained proto-policy library for zero-shot or few-shot transfer to entirely new tasks and integrating MAPS with Reinforcement Learning (RL) frameworks to allow robots to improve their skills beyond the initial expert demonstrations.

Summary updated

Abstract

Deep imitation learning requires many expert demonstrations, which can be hard to obtain, especially when many tasks are involved. However, different tasks often share similarities, so learning them jointly can greatly benefit them and alleviate the need for many demonstrations. But, joint multi-task learning often suffers from negative transfer, sharing information that should be task-specific. In this work, we introduce a method to perform multi-task imitation while allowing for task-specific features. This is done by using proto-policies as modules to divide the tasks into simple sub-behaviours that can be shared. The proto-policies operate in parallel and are adaptively chosen by a selector mechanism that is jointly trained with the modules. Experiments on different sets of tasks show that our method improves upon the accuracy of single agents, task-conditioned and multi-headed multi-task agents, as well as state-of-the-art meta learning agents. We also demonstrate its ability to autonomously divide the tasks into both shared and task-specific sub-behaviours.
BibTeX
@inproceedings{antotsiou2022modular,
  author = {Dafni Antotsiou and Carlo Ciliberto and Tae-Kyun Kim},
  title = {{Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=KZBRvt7PlB5},
}
Figure