Multi-Task Reinforcement Learning in Continuous Control with Successor Feature-Based Concurrent Composition

Yu Tang Liu, Aamir Ahmad

TL;DR

This paper introduces a novel framework for multi-task reinforcement learning in continuous control that unifies existing approaches (SF-GPI and value composition) to enable efficient policy composition from learned primitives. The method allows agents to learn new tasks by combining existing skills without retraining, demonstrating strong transfer learning capabilities and comparable single-task performance to state-of-the-art methods.

What Problem Does This Paper Address?

Reinforcement learning (RL) has shown great promise in solving complex robotic control tasks. However, a major hurdle for real-world applications is sample efficiency – RL agents often require vast amounts of data to learn a single task. This problem is exacerbated when dealing with multiple tasks, as traditional RL agents typically learn each task from scratch, failing to leverage prior knowledge. Multi-task RL aims to address this by enabling agents to learn multiple tasks simultaneously and transfer knowledge between them. Existing multi-task RL methods, particularly those based on successor features (SFs), have shown promise in improving sample efficiency. However, there's a need for frameworks that can effectively compose policies from learned "primitives" in a concurrent manner, especially for continuous control domains where online policy extraction can be computationally prohibitive.

Methods

The core contribution of this paper is a novel framework for concurrent composition of policies in continuous control RL. This framework unifies two existing multi-task RL paradigms: Successor Feature-Generalized Policy Improvement (SF-GPI) and Value Composition (VC).

Key methodological aspects include:

  • Unification of SF-GPI and VC: The authors bridge the gap between SF-GPI (which improves policies by selecting primitives with the highest associated action-value) and VC (which composes Q-functions by linearly combining constituent Q-functions).
  • Successor Feature-Based Composition: The framework leverages the compositional properties of successor features. Instead of composing Q-functions directly, it composes SFs, which are then used to derive policies. This allows for more flexible and efficient composition.
  • Concurrent Policy Composition: Unlike methods that compose primitives sequentially, this approach composes policies concurrently at each time step. This is crucial for real-time performance.
  • Novel Composition Operators: The paper introduces several composition operators:
  • Successor Feature-based Value Composition (SFV): Averages SFs of primitives to approximate the composite SF.
  • Maximum Successor Feature Composition (MSF): Selects the maximum SF across primitives for each sub-task.
  • Generalized Policy Improvement (GPI) Composition: A reformulation of SF-GPI for composition.
  • Multiplicative Compositional Policy (MCP): Extends multiplicative policy composition from imitation learning to RL for concurrent composition.
  • Multiplicative Compositional Action (MCA): A method for composing policies directly in the action space using an "impact matrix" derived from SFs.
  • Direct Action Composition (DAC) / DAC-GPI: Further action-space composition methods that leverage the impact matrix.
  • Impact Matrix: A novel component that quantifies the relationship between successor features and action components, enabling more informed action-space composition.
  • Training: The framework is trained end-to-end using a soft policy iteration approach, similar to standard RL, but applied to SFs and primitives.

Datasets

The paper introduces two new benchmark environments for multi-task continuous control:

  • Pointmass: A simulated environment where a robot needs to navigate to a target location.
  • Pointer: Another simulated environment involving robot control with thrust and attitude.

These environments are built using IsaacGym, which facilitates large-scale parallelization for accelerated training and experimentation. The environments are designed to test transfer learning capabilities by defining various feature sets (sub-tasks) and mixture tasks (combining different objectives).

Key Insights and Ideas

  • Compositional SFs are Key: The paper argues that composing successor features, rather than Q-functions directly, offers a more effective and flexible path to multi-task policy composition in continuous control.
  • Bridging Value and Policy Space: A significant insight is the derivation of how composition in the value space (SFs) translates to composition in the policy space, providing an analytical foundation for their methods.
  • Action-Space Composition is Feasible: The introduction of the "impact matrix" and MCA/DAC methods demonstrates that effective policy composition can be achieved directly in the action space, overcoming the intractability of direct policy extraction in continuous domains.
  • Noise Mitigation is Crucial: The paper highlights that composition noise, arising from redundant primitives or imperfect composition, can hinder training stability and performance. Methods like DAC are shown to effectively mitigate this noise.
  • Transferability is Achievable: The experimental results strongly suggest that agents trained with this framework can effectively transfer learned skills to new, unseen tasks, outperforming single-task RL agents in transfer scenarios.

Key Results

  • Comparable Single-Task Performance: The proposed multi-task agents achieve single-task performance on par with Soft Actor-Critic (SAC), a strong baseline.
  • Effective Transfer Learning: The agents demonstrate significant transfer learning capabilities to unseen mixture and position control tasks, often outperforming SAC in these transfer scenarios.
  • Improved Sample Efficiency: While not explicitly quantified with a direct comparison to single-task RL, the framework's design based on SFs and composition inherently aims to improve sample efficiency by reusing learned primitives.
  • Noise Reduction: DAC and DAC-GPI methods are shown to effectively mitigate composition noise, leading to better performance and stability compared to MSF and SF-GPI in certain scenarios.
  • Benchmark Environments: The introduction of Pointmass and Pointer environments provides valuable tools for future research in multi-task continuous control.

Limitations and Future Work

  • Composition Loss: The paper acknowledges that due to the imperfect nature of composition rules, composite policies may not achieve optimal performance, leading to a "composition loss" that can result in lower performance compared to single-task agents in some cases.
  • Impact Matrix Estimation: The estimation of the impact matrix can be noisy, especially in the early stages of training, which can limit the effectiveness of DAC methods. Averaging over primitives and temporal steps is used to mitigate this.
  • MSF Impracticality: The MSF composition is found to be impractical for a large number of primitives due to its optimistic linearity assumption and requirement for feature dimension to match the number of primitives.
  • Future Work: The authors suggest that it might be beneficial to initially train primitives using alternative methods before transitioning to DAC(-GPI) once SFs achieve satisfactory precision. Further research could explore more robust impact matrix estimation techniques and investigate the impact of different composition methods on training stability and performance. The potential for autotelic agents to set their own goals and curricula using this framework is also highlighted.

Summary updated

Abstract

Deep reinforcement learning (DRL) frameworks are increasingly used to solve high-dimensional continuous-control tasks in robotics. However, due to the lack of sample efficiency, applying DRL for online learning is still practically infeasible in the robotics domain. One reason is that DRL agents do not leverage the solution of previous tasks for new tasks. Recent work on multi-tasking DRL agents based on successor features has proven to be quite promising in increasing sample efficiency. In this work, we present a new approach that unifies two prior multi-task RL frameworks, SF-GPI and value composition, for the continuous control domain. We exploit compositional properties of successor features to compose a policy distribution from a set of primitives without training any new policy. Lastly, to demonstrate the multi-tasking mechanism, we present a new benchmark for multi-task continuous control environment based on Raisim. This also facilitates large-scale parallelization to accelerate the experiments. Our experimental results in the Pointmass environment show that our multi-task agent has single task performance on par with soft actor critic (SAC) and the agent can successfully transfer to new unseen tasks where SAC fails. We provide our code as open-source at https://github.com/robot-perception-group/concurrent_composition for the benefit of the community.
BibTeX
@misc{liu2023multi,
  author = {Yu Tang Liu and Aamir Ahmad},
  title = {{Multi-Task Reinforcement Learning in Continuous Control with Successor Feature-Based Concurrent Composition}},
  year = {2023},
  eprint = {2303.13935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.13935},
}
Figure