Abstract
Prompting has emerged as the dominant paradigm for adapting large, pre-trained transformer-based models to downstream tasks. The Prompting Decision Transformer (PDT) enables large-scale, multi-task offline reinforcement learning pre-training by leveraging stochastic trajectory prompts to identify the target task. However, these prompts are sampled uniformly from expert demonstrations, overlooking a critical limitation: Not all prompts are equally informative for differentiating between tasks. To address this, we propose an inference time bandit-based prompt-tuning framework that explores and optimizes trajectory prompt selection to enhance task performance. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration compared to prompt-tuning baselines.
Introduction
In the domain of foundation models, prompting is a critical mechanism for task adaptation. In Reinforcement Learning (RL), the Prompting Decision Transformer (PDT) utilizes segments of expert trajectories—sequences of states, actions, and rewards—as prompts to guide the agent toward a specific task. While effective, the standard PDT architecture relies on uniform random sampling of these expert segments. This approach is inherently flawed: in multi-task environments, many demonstrations are generic or uninformative, leading to significant performance degradation if a "poor" prompt is selected.
The Bandit-Based Prompt-Tuning Framework
To overcome the limitations of random sampling and the high computational cost of gradient-based prompt-tuning, we introduce a Contextual Multi-Armed Bandit (CMAB) approach. This framework optimizes prompt selection at inference time while keeping the underlying Transformer backbone frozen.
The bandit treats the selection of prompt segments as "arms." When a prompt is chosen, the PDT executes the task, and the resulting total reward (online return) serves as the feedback signal for the bandit. To ensure the search space remains manageable as prompt length increases, we employ a linear scaling strategy, maintaining independent reward models for each prompt position. We evaluated standard exploration strategies, including $\epsilon$-greedy and Upper Confidence Bounds (UCB), to balance the discovery of new informative prompts with the exploitation of known high-performers.
Experimental Setup
The framework was tested in a 2D point-mass environment where an agent must navigate to specific goal coordinates. We defined 60 distinct tasks by varying the goal's radius and angle. An offline dataset of 1 million steps per task was used to train the PDT, with the top 1% of trajectories forming the "expert" prompt library.
Key Results and Performance
Our experiments demonstrate that the bandit-based approach significantly improves both the speed of convergence and the final performance of the agent.
As shown in the results, the bandit method (UCB and $\epsilon$-greedy) achieves near-optimal returns within a few dozen rollouts. In contrast, gradient-based methods like ZO-RankSGD require significantly more interaction—often five times as many rollouts—to reach comparable performance. The bandit's ability to evaluate segments independently prevents "noisy" updates from degrading overall prompt quality, a common issue in methods that perturb the entire prompt vector simultaneously.
Spatial Awareness and Exploration
A key advantage of the CMAB approach is its efficient exploration of the prompt space. By visualizing the prompts selected over time, we can see how the bandit learns to prioritize "informative" demonstrations.
The bandit successfully learns to avoid prompts located near the starting position, which are often ambiguous across multiple tasks, and instead selects prompts that show the agent nearing the specific goal. This spatial awareness is a direct result of the bandit's reward-driven exploration.
Conclusion and Future Work
This work provides a lightweight and efficient alternative to traditional prompt-tuning. By framing prompt selection as a bandit problem, we achieve rapid adaptation in multi-task RL environments with minimal online interaction. Future research will focus on scaling this approach to high-dimensional robotics tasks and exploring the synthesis of continuous, synthetic prompts beyond the existing expert library.
Summary updated