Large Sequence Models for Sequential Decision-Making: A Survey

Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo Mai, Jun Wang, Haifeng Zhang, Weinan Zhang

TL;DR

This survey explores the burgeoning field of using large sequence models, particularly Transformers, for sequential decision-making (SDM) tasks. It highlights how these models address long-standing challenges in Reinforcement Learning (RL) like sample efficiency and credit assignment, and discusses future directions for building more capable and generalist decision-making agents.

What Problem Does This Paper Address?

Sequential decision-making (SDM) problems, common in areas like robotics and game playing, have traditionally been tackled with Deep Reinforcement Learning (DRL). However, DRL faces significant hurdles:

  • Sample Efficiency: DRL often requires vast amounts of interaction with an environment to learn effectively, which can be costly or impossible in real-world scenarios.
  • Credit Assignment: Determining which past actions led to a current reward or failure is challenging, especially over long sequences of decisions.
  • Partial Observability: Agents often don't have complete information about the environment's state, making decision-making more complex.

While large sequence models (like Transformers) have revolutionized Natural Language Processing (NLP) and Computer Vision (CV) with their ability to handle vast amounts of data and capture complex patterns, their application to SDM is a more recent and rapidly evolving area. This paper surveys how these powerful models are being adapted to overcome DRL's limitations and unlock new capabilities in SDM.

Methods

The paper surveys various approaches that leverage Transformer architectures for SDM. Key methods and concepts discussed include:

  • Transformer Architecture: The core of these models, utilizing self-attention mechanisms to capture long-range dependencies in sequences, overcoming the limitations of recurrent neural networks (RNNs).
  • Decision Transformer (DT): Frames RL as a sequence modeling problem, feeding sequences of states, actions, and desired returns to a Transformer to predict actions. This approach treats RL as a supervised learning problem.
  • Trajectory Transformer (TT): Models the entire trajectory as a sequence and learns a world model to predict future states and rewards, enabling planning.
  • Upside-Down Reinforcement Learning (UDRL): A paradigm that reverses the roles of actions and returns, treating RL as a supervised learning task by aiming to achieve specific desired returns.
  • Imitation Learning (IL) variants: Behavioral Cloning (BC) and Generative Adversarial Imitation Learning (GAIL) are discussed as ways to train agents from expert demonstrations.
  • Pre-training Strategies: Leveraging large datasets (often from NLP/CV) to pre-train models for general skills, which are then fine-tuned for specific downstream SDM tasks. This includes pre-training policies, reward functions, and world models.
  • Hybrid Parallelism: Techniques combining data, model, and pipeline parallelism to efficiently train extremely large Transformer models across multiple devices.

Datasets

The paper discusses the use of various datasets, but does not focus on specific benchmark datasets for evaluation. Instead, it highlights:

  • Offline Datasets: Static datasets collected from prior interactions, crucial for offline RL.
  • Massive Datasets: The paper emphasizes the trend towards using extremely large datasets (e.g., billions of tokens) for pre-training, drawing from diverse knowledge domains (language, vision, multi-modal).
  • Environment Simulators: For online RL, simulators are used to generate vast amounts of interaction data.

Key Insights and Ideas

  • Transformer as a Unified Framework: Transformers offer a powerful and scalable architecture that can unify various aspects of SDM, from policy learning to world modeling, and even bridge RL with supervised and self-supervised learning.
  • RL as Sequence Modeling: Reframing RL problems as sequence modeling tasks (e.g., Decision Transformer) simplifies training and allows leveraging the strengths of large pre-trained models, bypassing some traditional DRL challenges.
  • Pre-training for Generalization: Pre-training on diverse data (especially multi-modal) is key to developing agents with general skills and enabling few-shot adaptation to new tasks, reducing the need for extensive online fine-tuning.
  • Scaling Laws Apply to SDM: Similar to NLP and CV, increasing model size and data volume in SDM leads to predictable performance improvements, suggesting that larger, more capable decision models are achievable.
  • Addressing DRL's Core Challenges: Sequence models inherently handle long-term dependencies better than RNNs, aiding credit assignment and partial observability. Their supervised nature also improves sample efficiency compared to traditional trial-and-error DRL.

Key Results

  • Transformer-based methods have demonstrated strong performance and generalizability across various SDM tasks, often surpassing traditional RL methods, especially in non-Markovian environments.
  • Techniques like Decision Transformer and Trajectory Transformer have shown promise in effectively leveraging sequence modeling for offline RL and planning.
  • Pre-training strategies, inspired by NLP and CV, are proving effective in enabling few-shot learning and rapid adaptation for agents.
  • The paper highlights the trend towards scaling up models, with Transformer architectures showing advantages in parallelization and efficiency for large-scale training.

Limitations and Future Work

  • Theoretical Guarantees: Converting RL to sequence modeling can lead to a loss of theoretical guarantees for policy optimization, an area needing further research to integrate traditional RL theory.
  • Customized Architectures: While vanilla Transformers are used, there's ample room for developing specialized Transformer architectures tailored for SDM, incorporating RL-specific semantics and attention mechanisms.
  • Unified Frameworks: Developing a single, unified framework that can encompass various RL scenarios and tasks remains a significant research challenge.
  • Data Distribution Shifts: SDM problems are prone to severe distribution shifts due to the agent's actions affecting the environment. Developing algorithms that learn stably despite these shifts is crucial.
  • Efficient Training Systems: Training massive decision models requires significant computational resources. Developing efficient training systems, especially for handling large datasets and complex simulation environments, is an ongoing challenge.
  • Multi-modal Integration: Effectively incorporating multi-modal knowledge (vision, language) into SDM agents and achieving natural cross-modal alignment is an important future direction.
  • Automatic Resource Management: For large-scale multi-task pre-training, automated resource allocation for different environments and models is needed to overcome manual and often sub-optimal configurations.

Summary updated

Abstract

Transformer architectures have facilitated the development of large-scale and general-purpose sequence models for prediction tasks in natural language processing and computer vision, e.g., GPT-3 and Swin Transformer. Although originally designed for prediction problems, it is natural to inquire about their suitability for sequential decision-making and reinforcement learning problems, which are typically beset by long-standing issues involving sample efficiency, credit assignment, and partial observability. In recent years, sequence models, especially the Transformer, have attracted increasing interest in the RL communities, spawning numerous approaches with notable effectiveness and generalizability. This survey presents a comprehensive overview of recent works aimed at solving sequential decision-making tasks with sequence models such as the Transformer, by discussing the connection between sequential decision-making and sequence modeling, and categorizing them based on the way they utilize the Transformer. Moreover, this paper puts forth various potential avenues for future research intending to improve the effectiveness of large sequence models for sequential decision-making, encompassing theoretical foundations, network architectures, algorithms, and efficient training systems. As this article has been accepted by the Frontiers of Computer Science, here is an early version, and the most up-to-date version can be found at https://journal.hep.com.cn/fcs/EN/10.1007/s11704-023-2689-5
BibTeX
@article{wen2023large,
  author = {Muning Wen and Runji Lin and Hanjing Wang and Yaodong Yang and Ying Wen and Luo Mai and Jun Wang and Haifeng Zhang and Weinan Zhang},
  title = {{Large Sequence Models for Sequential Decision-Making: A Survey}},
  journal = {Frontiers of Computer Science},
  year = {2023},
  doi = {10.1007/s11704-023-2689-5},
}
Figure