Towards General-Purpose Model-Free Reinforcement Learning

Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, Michael Rabbat

TL;DR: Reinforcement learning algorithms are typically hyper-specialized for specific tasks, while general-purpose model-based methods are computationally expensive. This paper introduces MR.Q, a model-free RL algorithm that leverages model-based representation learning to achieve strong performance across 118 diverse environments using a single set of hyperparameters. It offers the sample efficiency and generality of model-based methods but with the lightweight speed and simplicity of model-free approaches.

Introduction

In theory, Reinforcement Learning (RL) is a universal framework: an agent learns optimal behavior given an objective and environment interactions. In practice, however, RL algorithms are highly specialized. Researchers typically need entirely different algorithms and heavily tuned hyperparameters depending on whether an environment uses continuous versus discrete actions, or pixel versus vector observations (e.g., using TD3 for robotics and Rainbow for Atari).

Recently, model-based RL methods like DreamerV3 and TD-MPC2 have shown that a single algorithm can master diverse benchmarks without retuning hyperparameters. However, these methods build complex world models to simulate rollouts and plan, making them computationally heavy, slow, and complex to implement. The authors ask: can we achieve the general-purpose power of model-based RL using a much simpler, faster, model-free approach?

The MR.Q Approach

The authors propose MR.Q (Model-based Representations for Q-learning). The core idea is to split the RL agent into two parts: an encoder and a decoupled RL backbone.

  1. Model-Based Representation Learning: The encoder maps raw states and actions into a unified embedding space. Instead of relying solely on standard RL reward signals, this encoder is trained using model-based objectives. It unrolls dynamics over a short horizon to predict future rewards, next-state embeddings, and episode terminations.
  2. Model-Free Reinforcement Learning: Once the states and actions are mapped into this abstract embedding space, MR.Q uses standard, lightweight model-free techniques (like TD3 for continuous actions or Gumbel-Softmax for discrete actions) to learn the value function and policy.

By mapping everything into a unified latent space, MR.Q strips away environment-specific quirks (like whether the input is an image or a vector), allowing a single set of hyperparameters to work everywhere. Furthermore, the authors provide theoretical proofs showing that if these embeddings can linearly represent the environment's dynamics and rewards, they can also represent the true value function.

Experimental Setup

The authors evaluated MR.Q across 4 popular RL benchmarks, encompassing 118 distinct environments:

  • Gym - Locomotion: 5 continuous control tasks using low-level vector states.
  • DeepMind Control (DMC) Suite - Proprioceptive: 28 continuous control robotics tasks using vector observations.
  • DMC Suite - Visual: The exact same 28 tasks, but using raw pixel images as observations.
  • Atari 10M: 57 classic arcade games featuring discrete actions and pixel observations.

Key Insights

  • Representations are the real MVP: The success of heavy model-based RL algorithms might actually stem from the rich, robust representations they learn by predicting dynamics and rewards, rather than the actual planning or simulated rollouts they perform. MR.Q proves you can throw away the planning step and still get top-tier results.
  • Decoupling stabilizes learning: By separating the representation learning (model-based) from the policy optimization (model-free), MR.Q avoids the non-stationarity issues that usually plague RL, where the target values are constantly shifting as the policy updates.
  • The "No Free Lunch" of RL Benchmarks: The authors observed a striking lack of transfer between benchmarks. Algorithms that dominate Gym often fail in DMC, and Atari specialists (like DreamerV3) outright fail on complex continuous control tasks like the DMC Humanoid. This highlights the danger of the community overfitting algorithms to single benchmarks.

Results

Comprehensive performance and efficiency comparison of MR.Q against baselines across diverse benchmarks.
  • General Performance: MR.Q achieves competitive or state-of-the-art performance across all four benchmarks using a single set of hyperparameters. It is the strongest overall method across the continuous control benchmarks (winning both DMC Proprioceptive and Visual).
  • Parameter Efficiency: MR.Q uses significantly fewer parameters than general model-based baselines. On Atari, MR.Q uses just 4.4M parameters, compared to DreamerV3's massive 187.3M parameter model.
  • Speed: Because it avoids planning and simulated rollouts, MR.Q is vastly faster. On the HalfCheetah environment, MR.Q trains at 49 frames per second (FPS) compared to DreamerV3's 18 FPS and TD-MPC2's 14 FPS. During evaluation, MR.Q is nearly 10x faster than DreamerV3.

Limitations and Future Work

The authors acknowledge that MR.Q is a first step and is not equipped to handle hard exploration tasks or non-Markovian environments (where the current state doesn't contain all the information needed to make a decision). Additionally, while MR.Q performs exceptionally well on standard RL benchmarks, established algorithms like PPO have proven their versatility in wildly unique, complex settings (like training Large Language Models or drone racing). Future work will need to test if lightweight, general-purpose algorithms like MR.Q can scale to those highly complex, real-world applications.

Summary updated

Abstract

Reinforcement learning (RL) promises a framework for near-universal problem-solving. In practice however, RL algorithms are often tailored to specific benchmarks, relying on carefully tuned hyperparameters and algorithmic choices. Recently, powerful model-based RL methods have shown impressive general results across benchmarks but come at the cost of increased complexity and slow run times, limiting their broader applicability. In this paper, we attempt to find a unifying model-free deep RL algorithm that can address a diverse class of domains and problem settings. To achieve this, we leverage model-based representations that approximately linearize the value function, taking advantage of the denser task objectives used by model-based RL while avoiding the costs associated with planning or simulated trajectories. We evaluate our algorithm, MR.Q, on a variety of common RL benchmarks with a single set of hyperparameters and show a competitive performance against domain-specific and general baselines, providing a concrete step towards building general-purpose model-free deep RL algorithms.
BibTeX
@inproceedings{fujimoto2025towards,
  author = {Scott Fujimoto and Pierluca D'Oro and Amy Zhang and Yuandong Tian and Michael Rabbat},
  title = {{Towards General-Purpose Model-Free Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=R1hIXdST22},
}
Figure