Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-task Offline RL

Finn Rietz, Oleg Smirnov, Sara Karimi, Lele Cao

Abstract

Prompting has emerged as the dominant paradigm for adapting large, pre-trained transformer-based models to downstream tasks. The Prompting Decision Transformer (PDT) enables large-scale, multi-task offline Reinforcement Learning (RL) pre-training by leveraging stochastic trajectory prompts to identify the target task. However, these prompts are sampled uniformly from expert demonstrations, overlooking a critical limitation: not all prompts are equally informative for differentiating between tasks. This limits generalization and adaptation, especially in low-data or open-world settings where sample efficiency is crucial. To address this issue, we propose a lightweight, inference-time, bandit-based prompt-tuning framework. The bandit explores and optimizes trajectory prompt selection to enhance task performance, while avoiding costly fine-tuning of the transformer backbone. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration compared to prompt-tuning baselines. These results highlights the importance of adaptive prompt selection mechanisms for efficient generalization in offline multi-task RL.

Introduction

Prompting Decision Transformers (PDTs) use trajectory prompts—short sequences of expert demonstrations—to adapt to new tasks without parameter fine-tuning. However, the standard practice of sampling these prompts uniformly at random often yields uninformative context, such as segments where an agent is far from a goal. This uninformed sampling creates a performance bottleneck, particularly in few-shot scenarios where sample efficiency is paramount.

Methodology: Bandit-Based Prompt Selection

We reframe prompt selection as a Contextual Multi-Armed Bandit (CMAB) problem that operates at inference time. Instead of fine-tuning the massive transformer backbone, the system uses a bandit to learn which prompt segments yield the highest return. To avoid the combinatorial explosion of searching through all possible prompt combinations, our architecture employs $J$ independent arms, where $J$ is the number of segments in the prompt.

The bandit maintains a reward model for each segment position, treating the segments as context to estimate the expected RL return. During online rollouts, the bandit selects prompts using exploration strategies like $\epsilon$-greedy or Upper Confidence Bound (UCB). The resulting environment score is then used as a reward signal to update the bandit's selection policy.

Experimental Results

We evaluated our framework on a 2D planar point-agent navigation task involving 60 distinct goal-based tasks. The results demonstrate that the bandit-based approach significantly outperforms standard PDT and optimization baselines like Gaussian hill climbing or Zeroth-Order SGD (ZO-RankSGD).

Comparison of total reward across different prompt segment counts (J=1, 2, 4) for various tuning methods.

As shown in the performance plots, the bandit methods achieve near-optimal returns within the first few dozen online rollouts. In contrast, standard PDT with random sampling fails to reach optimal performance. Furthermore, our method scales robustly as the prompt size increases, whereas perturbation-based baselines often degrade due to the increased complexity of the search space.

Emergent Spatio-Temporal Awareness

A key insight from our experiments is that the bandit naturally discovers which information is most relevant for task identification. Visualizations of the exploration process show that while the bandit initially samples prompts from across the state space, it quickly learns to prioritize segments located physically closer to the goal states.

Visualization of prompt exploration in the 2D environment, showing how the CMAB approach focuses on goal-relevant regions over time compared to baselines.

Unlike global perturbation methods that add noise to the entire prompt—often destroying informative segments in the process—the bandit evaluates segments independently. This allows it to preserve high-performing segments while iteratively swapping out less informative ones, leading to superior sample complexity and faster convergence.

Conclusion

This work demonstrates that a smart model is only as good as its prompt. By optimizing prompt selection at inference time using a bandit framework, we can unlock the latent capabilities of pre-trained models for efficient multi-task generalization. Future work will focus on extending this method to high-dimensional state spaces and more complex robotic control environments.

Summary updated

Abstract

Prompting has emerged as the dominant paradigm for adapting large, pre-trained transformer-based models to downstream tasks. The Prompting Decision Transformer (PDT) enables large-scale, multi-task offline Reinforcement Learning (RL) pre-training by leveraging stochastic trajectory prompts to identify the target task. However, these prompts are sampled uniformly from expert demonstrations, overlooking a critical limitation: not all prompts are equally informative for differentiating between tasks. This limits generalization and adaptation, especially in low-data or open-world settings where sample efficiency is crucial. To address this issue, we propose a lightweight, inference-time, bandit-based prompt-tuning framework. The bandit explores and optimizes trajectory prompt selection to enhance task performance, while avoiding costly fine-tuning of the transformer backbone. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration compared to prompt-tuning baselines. These results highlights the importance of adaptive prompt selection mechanisms for efficient generalization in offline multi-task RL.
BibTeX
@misc{rietz2025prompt,
  author = {Finn Rietz and Oleg Smirnov and Sara Karimi and Lele Cao},
  title = {{Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-task Offline RL}},
  year = {2025},
  eprint = {2502.06358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.06358},
}
Figure