Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RL

Finn Rietz, Oleg Smirnov, Sara Karimi, Lele Cao

TL;DR

This paper introduces a lightweight, inference-time framework that uses contextual multi-armed bandits to optimize prompt selection for Prompting Decision Transformers (PDT) in multi-task offline reinforcement learning. By actively exploring and selecting the most informative expert trajectory segments instead of sampling them uniformly, the method significantly boosts few-shot adaptation and sample efficiency without requiring costly fine-tuning of the underlying transformer backbone.

What Problem Does This Paper Address?

In multi-task offline Reinforcement Learning (RL), models like the Prompting Decision Transformer (PDT) use "prompts"—short segments of expert demonstrations—to identify which task they are supposed to execute. However, standard PDT architectures sample these prompts uniformly at random from a dataset.

The core issue is that not all prompts are equally informative. In complex or open-world settings, some trajectory segments might look identical across multiple tasks, leading to ambiguity, poor task identification, and degraded performance. While some researchers have tried to optimize these prompts using gradient-based methods, doing so often violates the causal structure of the prompt tokens, fails in discrete settings, or requires expensive online data collection. The authors set out to find a sample-efficient way to find the best possible prompts at inference time without altering the pre-trained transformer's weights.

Methods

The authors reframe prompt selection as a Contextual Multi-Armed Bandit (CMAB) problem.

If a prompt consists of $J$ different segments, a naive bandit approach would treat every possible combination of segments as an arm, leading to a combinatorial explosion. To keep the approach scalable, the authors propose a architecture with exactly $J$ arms—one for each position in the prompt.

Here is how it works at inference time:

  1. Reward Modeling: The bandit maintains a separate reward model for each position $j$ in the prompt. It treats the candidate expert trajectory segments as the "context."
  2. Prompt Construction: At each evaluation round, the bandit predicts how well each segment will perform in each position, generating a prediction matrix.
  3. Exploration vs. Exploitation: Using standard bandit strategies like $\epsilon$-greedy or Upper Confidence Bounds (UCB), it selects a combination of segments to form a prompt.
  4. Evaluation: The prompt is prepended to the PDT, an online rollout is performed in the environment, and the resulting total return is fed back to the bandit as a reward signal to update its models.

This setup ensures that the search space scales linearly, rather than combinatorially, with the number of prompt segments.

Datasets

The framework was evaluated using a controlled 2D proof-of-concept environment featuring a planar 2D point agent tasked with reaching specific goal coordinates.

The 2D multi-task environment setup
  • Tasks: 60 distinct tasks parameterized by the goal's radius and angle.
  • Offline Dataset: Generated by training a Proximal Policy Optimization (PPO) agent for 1 million steps on each task. The top percentile of these trajectories was set aside to serve as the expert demonstrations ($P_i$) from which the prompts were sampled.

Key Insights and Ideas

  • The Pitfall of Global Perturbations: Traditional optimization baselines (like Gaussian hill climbing or Zeroth-Order Stochastic Gradient Descent) perform poorly because they inject noise into the entire prompt at once. This unnecessarily disrupts segments that are already highly informative.
  • Independent Position Tracking: By treating each slot in the prompt sequence as an independent arm, the bandit can preserve high-performing segments in one part of the prompt while actively exploring and optimizing other slots.
  • Spatio-Temporal Awareness: Visualization of the prompt space reveals that the bandit quickly learns to avoid ambiguous, low-performance prompts (such as trajectories near the starting center) and prioritizes trajectories closer to the actual goal states.

Key Results

The bandit-based prompt-tuning approach was benchmarked against standard PDT (no tuning), Gaussian perturbation hill climbing, and ZO-RankSGD across different prompt lengths ($J \in \{1, 2, 4\}$).

Fig. 2: Inference time performance gains due to prompt-tuning over 250 episodes. ZO-RankSGD performs a total of m = 5 × 250 rollouts to estimate the prompt gradient 250 times, which we squash into the 0 - 250 range in the plot. Results are averaged over all training tasks and three seeds. To increase readability, the shaded area corresponds only to 0.25 standard deviations around the mean.
  • Rapid Convergence: As shown in Figure 2, both the $\epsilon$-greedy and UCB bandit strategies achieved near-optimal rewards within the first few online rollouts, drastically outperforming the gradient-free optimization baselines.
  • Scalability with Prompt Size: While standard PDT's performance naturally improves with larger prompt sizes (as random sampling is more likely to hit a good segment), perturbation-based baselines actually degraded because the larger prompt space meant injecting more disruptive noise. The bandit approach maintained top-tier performance regardless of prompt size.

Limitations and Future Work

The primary limitation acknowledged by the authors is that the framework has only been validated within a simplified 2D proof-of-concept environment. While this environment successfully demonstrates the mechanics of prompt informativeness and bandit exploration, it remains to be seen how well the linear arm-decomposition scales to highly complex, high-dimensional control tasks (e.g., MuJoCo or Atari benchmarks). The authors note that ongoing work is focused on extending this method to these more challenging environments.

Summary updated

Abstract

Prompting has emerged as the dominant paradigm for adapting large, pre-trained transformer-based models to downstream tasks. The Prompting Decision Transformer (PDT) enables large-scale, multi-task offline Reinforcement Learning (RL) pre-training by leveraging stochastic trajectory prompts to identify the target task. However, these prompts are sampled uniformly from expert demonstrations, overlooking a critical limitation: not all prompts are equally informative for differentiating between tasks. This limits generalization and adaptation, especially in low-data or open-world settings where sample efficiency is crucial. To address this issue, we propose a lightweight, inference-time, bandit-based prompt-tuning framework. The bandit explores and optimizes trajectory prompt selection to enhance task performance, while avoiding costly fine-tuning of the transformer backbone. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration compared to prompt-tuning baselines. These results highlights the importance of adaptive prompt selection mechanisms for efficient generalization in offline multi-task RL.
BibTeX
@misc{rietz2025prompt,
  author = {Finn Rietz and Oleg Smirnov and Sara Karimi and Lele Cao},
  title = {{Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RL}},
  year = {2025},
  eprint = {2502.06358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.06358},
}
Figure