Modular Adaptive Policy Selection for Multi- Task Imitation Learning through Task Division

Dafni Antotsiou, Carlo Ciliberto, Tae-Kyun Kim

TL;DR

MAPS is a novel framework for multi-task imitation learning that decomposes complex robot tasks into shareable "proto-policies." By using an adaptive selector to combine these modules, the system achieves significant performance gains over state-of-the-art meta-learning methods while effectively avoiding the "negative transfer" that often plagues multi-task systems.

What Problem Does This Paper Address?

Imitation Learning (IL)—where a robot learns by mimicking human experts—is data-hungry. Learning ten different tasks usually requires ten times the demonstrations. Multi-Task Learning (MTL) aims to solve this by training one model on everything, hoping that "positive transfer" (knowledge from one task helping another) will reduce the data needed.

However, MTL often hits a wall called negative transfer: when tasks are too different, forcing them into the same neural representation actually hurts performance. For example, the motor skills needed to "push a button" might interfere with the skills needed to "open a window." Current approaches like MAML (Meta-Learning) or task-conditioned networks often struggle to balance these conflicting requirements, especially as tasks become more complex.

Methods

The authors introduce Modular Adaptive Policy Selection (MAPS). Instead of one monolithic policy, MAPS uses a library of parallel proto-policies (modules).

  1. The Modules: Each module is a small neural network that learns a simple, primitive behavior (like "reaching" or "grasping"). Crucially, these modules only see the environment state, not the task ID, forcing them to learn general-purpose physical maneuvers.
  2. The Selector (G): This is the "brain" of the operation. It looks at both the current state and the specific task ID to decide which modules to activate and how much to weight their contributions.
  3. The Loss Function: This is the "secret sauce." To make the selector work, the authors designed a composite loss function with four specific goals:
  4. Sharing: Encourages different tasks to use the same modules for similar states.
  5. Exploration: Prevents the system from relying on just one or two "super-modules," ensuring all modules learn something useful.
  6. Sparsity: Forces the selector to pick only a few modules at a time, making the behaviors more distinct.
  7. Smoothness: Ensures the robot doesn't jitter between different modules mid-action.

Datasets

The researchers evaluated MAPS on three distinct types of variation:

  • HalfCheetah Gravity: 5 tasks where the same robot runs under different gravity levels (environmental variation).
  • HalfCheetah Modified: 11 tasks where the robot's body parts (thigh, foot, etc.) are resized (model variation).
  • MetaWorld MT10: A popular benchmark of 10 diverse manipulation tasks (reach, push, pick-and-place, etc.) using a Sawyer robot arm (task variation).

Key Insights and Ideas

  • Task Decomposition is Automatic: The system doesn't need to be told how to break a task down. It autonomously discovers that "opening a drawer" and "opening a window" both share a "grab handle" sub-behavior.
  • Efficiency through Modularity: In the MT10 benchmark, MAPS with only 5 modules performed comparably to 10 separate agents. This suggests that complex task spaces have a much lower "intrinsic dimensionality" than we might think.
  • The Selector as a Regularizer: By balancing the four selector losses (sharing vs. sparsity), the model naturally finds the sweet spot between general-purpose skills and task-specific expertise, effectively neutralizing negative transfer.

Key Results

  • Superior Accuracy: MAPS consistently outperformed single-agent baselines, task-conditioned networks, and MAML. In many cases, it was over 10% more successful than the next best method.
  • Positive Transfer: MAPS performed better than single-agent baselines more than 90% of the time, proving that it successfully harvests knowledge across tasks.
  • Qualitative Proof: The authors visualized module usage and found that specific modules indeed learned distinct physical primitives. For instance, one module learned to "move forward," while another learned to "move backward," and the selector swapped between them depending on whether the robot was opening or closing a drawer.

Limitations and Future Work

While MAPS is powerful, it requires careful tuning of the $\lambda$ hyperparameters that balance the four selector losses. If sparsity is too high, the model becomes a collection of independent agents; if sharing is too high, it suffers from negative transfer.

The authors suggest several exciting next steps:

  • Transfer Learning: Using the pre-learned "library" of proto-policies to learn entirely new tasks with even fewer demonstrations.
  • Integration with RL: Moving beyond imitation (Behavioral Cloning) to Reinforcement Learning (like GAIL) to allow the robot to improve beyond the expert's performance.
  • Hierarchical Decomposition: Combining MAPS with hierarchical RL to handle even longer, more complex sequences of actions.

Summary updated

Abstract

Deep imitation learning requires many expert demonstrations, which can be hard to obtain, especially when many tasks are involved. However, different tasks often share similarities, so learning them jointly can greatly benefit them and alleviate the need for many demonstrations. But, joint multi-task learning often suffers from negative transfer, sharing information that should be task-specific. In this work, we introduce a method to perform multi-task imitation while allowing for task-specific features. This is done by using proto-policies as modules to divide the tasks into simple sub-behaviours that can be shared. The proto-policies operate in parallel and are adaptively chosen by a selector mechanism that is jointly trained with the modules. Experiments on different sets of tasks show that our method improves upon the accuracy of single agents, task-conditioned and multi-headed multi-task agents, as well as state-of-the-art meta learning agents. We also demonstrate its ability to autonomously divide the tasks into both shared and task-specific sub-behaviours.
BibTeX
@inproceedings{antotsiou2022modular,
  author = {Dafni Antotsiou and Carlo Ciliberto and Tae-Kyun Kim},
  title = {{Modular Adaptive Policy Selection for Multi- Task Imitation Learning through Task Division}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=KZBRvt7PlB5},
}

Loading related content...

Figure