Expressive Modeling Is Insufficient for Offline RL: A Tractable Inference Perspective

Xuejie Liu, Anji Liu, Guy Van den Broeck, Yitao Liang

Abstract

A popular paradigm for offline Reinforcement Learning (RL) tasks is to first fit the offline trajectories to a sequence model, and then prompt the model for actions that lead to high expected return. In addition to obtaining accurate sequence models, this paper highlights that tractability, the ability to exactly and efficiently answer various probabilistic queries, plays an important role in offline RL. Specifically, due to the fundamental stochasticity from the offline data-collection policies and the environment dynamics, highly non-trivial conditional/constrained generation is required to elicit rewarding actions. While it is still possible to approximate such queries, we observe that such crude estimates significantly undermine the benefits brought by expressive sequence models. To overcome this problem, this paper proposes Trifle (Tractable Inference for Offline RL), which leverages modern Tractable Probabilistic Models (TPMs) to bridge the gap between good sequence models and high expected returns at evaluation time. Empirically, Trifle achieves the most state-of-the-art scores in 9 Gym-MuJoCo benchmarks against strong baselines. Further, owing to its tractability, Trifle significantly outperforms prior approaches in stochastic environments and safe RL tasks with minimum algorithmic modifications.

Introduction

In offline Reinforcement Learning (RL), the "Sequence Modeling" paradigm (often referred to as Reinforcement Learning as Sequence Modeling or RvS) has gained significant traction. By treating RL as a sequence completion task, researchers have successfully applied large-scale architectures like Transformers to offline trajectory data. However, this paper argues that high model expressivity—the ability to fit complex data distributions—is insufficient for optimal performance. The critical bottleneck is often tractability: the ability to efficiently and exactly query the model during inference to identify actions that maximize expected returns.

The Challenge of Inference-time Suboptimality

Offline RL models frequently encounter "inference-time suboptimality." Even when a model has accurately learned the underlying data distribution, it may fail to select optimal actions during evaluation. This failure typically stems from two sources:

  1. Stochasticity: In environments with random dynamics, calculating the expected return requires averaging over numerous potential future states. Standard autoregressive models struggle with this marginalization on the fly.
  2. Complex Constraints: Selecting actions that are both high-reward and safety-compliant (e.g., torque limits) requires constrained generation, which is computationally difficult for standard sequence models.

Trifle: Leveraging Tractable Probabilistic Models

To address these challenges, the authors introduce Trifle (Tractable Inference for Offline RL). Trifle moves beyond standard neural architectures by incorporating Probabilistic Circuits (PCs).

Structure of a Probabilistic Circuit (PC) using sum and product nodes to represent distributions.

Probabilistic Circuits

PCs are computational graphs composed of "sum" and "product" nodes. Unlike standard neural networks, PCs represent probability distributions in a manner that allows for the exact calculation of marginals and conditionals in linear time relative to the circuit size. This mathematical property enables Trifle to perform the complex "what-if" reasoning required for RL without relying on high-variance Monte Carlo approximations.

The Trifle Framework

Trifle enhances action selection by adding a "correction term" derived from the PC. This term biases sampling toward actions with a high probability of achieving a target return. The framework includes two variants:

  • s-Trifle: Utilizes a single-step value estimate.
  • m-Trifle: Leverages the PC’s ability to marginalize over future states, enabling accurate multi-step lookahead in stochastic environments.

Experimental Evaluation

Trifle was evaluated across several benchmarks, including Gym-MuJoCo (D4RL), stochastic grid-worlds (Taxi and FrozenLake), and action-space-constrained MuJoCo environments.

Key Results

Trifle demonstrated superior performance across all categories:

  • State-of-the-Art Performance: Trifle achieved the highest average scores across 9 Gym-MuJoCo benchmarks, setting 7 new SOTA records.
  • Robustness to Stochasticity: In the Stochastic Taxi environment, m-Trifle significantly outperformed the Trajectory Transformer (TT), achieving a score of -57 compared to TT's -182.
  • Safety and Constraints: In constrained environments, Trifle's ability to condition action choices on safety limits in real-time allowed it to significantly outperform baselines.
Comparison of inference-time optimality scores across different benchmarks for DT, TT, and Trifle.

Conclusion

The success of Trifle underscores that for offline RL, the ability to query a model is as important as the model's capacity to learn. By bridging the gap between expressive modeling and tractable inference, TPMs like Probabilistic Circuits offer a promising path toward more reliable and safer RL agents in complex, real-world environments. Future work will focus on optimizing the computational execution of these models to match the speed of standard deep learning frameworks.

Summary updated

Abstract

A popular paradigm for offline Reinforcement Learning (RL) tasks is to first fit the offline trajectories to a sequence model, and then prompt the model for actions that lead to high expected return. In addition to obtaining accurate sequence models, this paper highlights that tractability, the ability to exactly and efficiently answer various probabilistic queries, plays an important role in offline RL. Specifically, due to the fundamental stochasticity from the offline data-collection policies and the environment dynamics, highly non-trivial conditional/constrained generation is required to elicit rewarding actions. it is still possible to approximate such queries, we observe that such crude estimates significantly undermine the benefits brought by expressive sequence models. To overcome this problem, this paper proposes Trifle (Tractable Inference for Offline RL), which leverages modern Tractable Probabilistic Models (TPMs) to bridge the gap between good sequence models and high expected returns at evaluation time. Empirically, Trifle achieves the most state-of-the-art scores in 9 Gym-MuJoCo benchmarks against strong baselines. Further, owing to its tractability, Trifle significantly outperforms prior approaches in stochastic environments and safe RL tasks (e.g. with action constraints) with minimum algorithmic modifications.
BibTeX
@inproceedings{liu2024expressive,
  author = {Xuejie Liu and Anji Liu and Guy Van den Broeck and Yitao Liang},
  title = {{Expressive Modeling Is Insufficient for Offline RL: A Tractable Inference Perspective}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=UZIHW8eFRp},
}
Figure