Test-time Offline Reinforcement Learning on Goal-related Experience

Marco Bagatella, Mert Albaba, Jonas Hübotter, Georg Martius, Andreas Krause

Abstract

Universal policies in offline goal-conditioned reinforcement learning (GCRL) often suffer from systematic underfitting because they are trained to perform well on average across a vast distribution of goals. This paper introduces Goal-Conditioned Test-Time Training (GC-TTT), a method that specializes a frozen pre-trained policy during inference. By filtering the offline dataset for experience that is both relevant to the agent's current state and optimal for the specific test-time goal, GC-TTT achieves substantial performance improvements across navigation and manipulation benchmarks.

Introduction

In standard offline GCRL, agents are trained on fixed datasets to reach arbitrary goals. At test time, these agents are typically evaluated using frozen weights. However, these "one-size-fits-all" policies often fail on specific tasks because the training objective prioritizes broad coverage over local precision. While Test-Time Training (TTT) has revolutionized fields like computer vision and natural language processing, its application in RL is hindered by the risk of imitating suboptimal or irrelevant data. GC-TTT addresses this by answering: how can we safely adapt a frozen RL policy at test time to excel at the specific goal it was just handed?

Methodology: Goal-Conditioned Test-Time Training

GC-TTT employs a receding-horizon approach to dynamically fine-tune the policy during evaluation. The process involves two critical components: data selection and periodic weight updates.

Data Selection

To ensure safe and effective adaptation, GC-TTT filters the pre-training dataset based on two criteria:

  1. Relevance: The algorithm selects sub-trajectories starting near the agent's current state, utilizing a distance function or a learned quasimetric.
  2. Optimality: Selected trajectories are further filtered to retain only those most likely to reach the target goal, estimated via an $H$-step return calculation using the pre-trained critic.
Conceptual overview of GC-TTT: specializing the policy using goal-related experience from the pre-training data.

Receding Horizon Training

The policy weights are reset to their pre-trained state every $K$ steps. The model then undergoes a few gradient steps on the filtered data before being rolled out in the environment. This cycle repeats, allowing the policy to adapt as the agent's state and proximity to the goal change.

Experimental Results

The authors evaluated GC-TTT using the OGBench suite, covering loco-navigation (PointMaze, AntMaze, HumanoidMaze) and robotic manipulation tasks.

Performance comparison showing average success rates for various baselines with and without GC-TTT.

The results demonstrate that GC-TTT provides massive boosts to both weak and strong baselines. For instance, Behavior Cloning (GC-BC) on the pointmaze expert task saw success rates jump from 5% to 81%. As shown in the aggregate results, even for state-of-the-art hierarchical algorithms like SAW, GC-TTT improved the average success rate from 86% to 92%.

Key Insights

  • Underfitting in Offline RL: The dramatic improvement from minimal test-time fine-tuning suggests that current offline RL algorithms significantly underfit the training data.
  • Synergy of Relevance and Optimality: Ablation studies show that filtering for only one of these criteria results in marginal gains; both are necessary for high performance.
  • Compute Efficiency: GC-TTT outperforms baseline models that were scaled up to match the same total inference compute (FLOPs), indicating that test-time adaptation is a more efficient use of resources than increasing parameter count.

Conclusion and Future Work

GC-TTT proves that test-time specialization is a powerful tool for offline RL, enabling agents to overcome the limitations of universal pre-training. While the current implementation maintains a control frequency of ~10 Hz, future work could focus on optimizing the data retrieval process for high-frequency robotics. Additionally, the principles of GC-TTT could be extended to online RL or reasoning tasks in large-scale foundation models.

Summary updated

BibTeX
@inproceedings{bagatella2026test,
  author = {Marco Bagatella and Mert Albaba and Jonas Hübotter and Georg Martius and Andreas Krause},
  title = {{Test-time Offline Reinforcement Learning on Goal-related Experience}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=NN4DoPVYT2},
}
Figure