Generative Decision Making Under Uncertainty

Aditya GroverAAAI 2023 · OpenReview

Abstract

In the fields of natural language processing (NLP) and computer vision (CV), recent advances in generative modeling have led to powerful machine learning systems that can effectively learn from large labeled and unlabeled datasets. These systems, by and large, apply a uniform pretrain-finetune pipeline on sequential data streams and have achieved state-of-the-art-performance across many tasks and benchmarks. In this talk, we present recent algorithms that extend this paradigm to sequential decision making, by casting it as an inverse problem that can be solved via deep generative models. These generative approaches are stable to train, provide a flexible interface for single- and multi-task inference, and generalize exceedingly well outside their training datasets. We instantiate these algorithms in the context of reinforcement learning and black-box optimization. Empirically, we demonstrate that these approaches perform exceedingly well on high-dimensional benchmarks outperforming the current state-of-the-art approaches based on forward models.

Introduction

The "pretrain-then-finetune" paradigm has fundamentally transformed Natural Language Processing (NLP) and Computer Vision (CV). By training large-scale models on vast datasets and subsequently adapting them to specific tasks, researchers have achieved unprecedented performance. However, translating this success to interactive decision-making domains—specifically Reinforcement Learning (RL) and Black-Box Optimization (BBO)—has historically been fraught with challenges.

Traditional approaches in these fields rely heavily on "forward models." In RL, these models (such as Q-learning) attempt to predict the expected value of a specific action. In BBO, they attempt to predict the output of an unknown objective function given a set of inputs. While intuitive, forward models often struggle with distribution shift, where the model encounters data or desired outcomes significantly different from those seen during training. This often leads to instability and a failure to generalize to high-performance targets.

The Shift to Inverse Generative Models

This research advocates for a paradigm shift: moving from forward models to inverse generative models. Instead of predicting the value of an action, inverse models ask: "What action or input is required to achieve a specific, desired outcome?"

By casting decision-making as an inverse problem, we can leverage the power of deep generative architectures like Transformers and Diffusion models. This approach offers several distinct advantages:

  1. Handling Multi-modality: A single desired outcome (e.g., winning a game) can often be achieved through multiple distinct action sequences. Generative models are naturally suited to modeling these complex, one-to-many relationships.
  2. Supervised Stability: Unlike traditional RL, which often relies on "bootstrapping" (updating estimates based on other estimates), inverse models can be trained using stable, vanilla supervised learning objectives.
  3. Robust Generalization: These models demonstrate a remarkable ability to generalize to "out-of-distribution" (OOD) targets, often identifying strategies to achieve scores higher than any single trajectory present in the training data.

Methodological Framework

The proposed framework is instantiated through several key techniques designed to handle sequential and high-dimensional data.

Decision Transformers (DT)

The Decision Transformer treats reinforcement learning as a sequence modeling problem. By utilizing an autoregressive Transformer architecture, the model maps a sequence of past states and desired "returns-to-go" (the remaining reward the agent aims to collect) directly to actions. This allows the agent to leverage long-range dependencies, which is crucial in environments with sparse rewards where the consequences of an action may not be apparent for many steps.

Generative Black-Box Optimization

In BBO, the goal is to find inputs that maximize an unknown function. This research utilizes autoregressive Transformers and Diffusion models to map high function values back to the input space. By doing so, the model "imagines" the design parameters required to achieve optimal results. To enhance this, novel forms of attention and guidance are employed to help the model extrapolate beyond the best examples in the training set.

Adaptation and Finetuning

To bridge the gap between offline pretraining and online execution, the framework introduces entropy-based regularization and hindsight relabeling. These methods allow the pretrained models to adapt efficiently to new data, ensuring they remain performant even as the environment or task specifications evolve.

Empirical Evaluation and Results

The effectiveness of generative decision-making is evaluated across diverse benchmarks, including large off-policy RL datasets and complex BBO "Design Benchmarks" spanning biology, sustainability, and engineering.

Key findings include:

  • Superior Performance: Generative approaches consistently outperform state-of-the-art forward models in high-dimensional tasks.
  • Long-Range Reasoning: Decision Transformers excel in tasks requiring memory of distant events, outperforming traditional temporal difference (TD) learning methods.
  • Scientific Design: In BBO contexts, the generative approach achieves top-tier results in complex design tasks, demonstrating its utility in real-world scientific discovery.

Conclusion and Future Frontiers

Generative decision-making represents a robust alternative to traditional predictive modeling. By treating trajectories as sequences and optimization as an inverse problem, we can build agents that are more stable, flexible, and capable of generalizing to novel scenarios.

Future work will focus on several critical frontiers:

  • Multi-task Scaling: Developing models capable of handling a wide array of varying task specifications simultaneously.
  • Pareto Efficiency: Navigating competing objectives, such as balancing performance with safety or resource constraints.
  • Integrated Exploration: Better incorporating active exploration strategies into the generative framework to allow agents to safely discover new behaviors in entirely unknown environments.

Summary updated

Abstract

In the fields of natural language processing (NLP) and computer vision (CV), recent advances in generative modeling have led to powerful machine learning systems that can effectively learn from large labeled and unlabeled datasets. These systems, by and large, apply a uniform pretrain-finetune pipeline on sequential data streams and have achieved state-of-the-art-performance across many tasks and benchmarks. In this talk, we will present recent algorithms that extend this paradigm to sequential decision making, by casting it as an inverse problem that can be solved via deep generative models. These generative approaches are stable to train, provide a flexible interface for single- and multi-task inference, and generalize exceedingly well outside their training datasets. We instantiate these algorithms in the context of reinforcement learning and black-box optimization. Empirically, we demonstrate that these approaches perform exceedingly well on high-dimensional benchmarks outperforming the current state-of-the-art approaches based on forward models.
BibTeX
@inproceedings{grover2023generative,
  author = {Aditya Grover},
  title = {{Generative Decision Making Under Uncertainty}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=fFC46nPFg5},
}
Figure