Abstract
Reinforcement learning (RL) has drawn increasing interests in recent years due to its tremendous success in various applications. However, standard RL algorithms can only be applied for single reward function, and cannot adapt to an unseen reward function quickly. In this paper, we advocate a general operator view of reinforcement learning, which enables us to directly approximate the operator that maps from reward function to value function. The benefit of learning the operator is that we can incorporate any new reward function as input and attain its corresponding value function in a zero-shot manner. To approximate this special type of operator, we design a number of novel operator neural network architectures based on its theoretical properties. Our design of operator networks outperform the existing methods and the standard design of general purpose operator network, and we demonstrate the benefit of our operator deep Q-learning framework in several tasks including reward transferring for offline policy evaluation (OPE) and reward transferring for offline policy optimization in a range of tasks.
1. Introduction
In standard Reinforcement Learning (RL), agents are typically trained to master a specific task defined by a fixed reward function. However, real-world applications often require agents to adapt to changing goals. If a reward function is modified, standard agents usually require complete retraining from scratch. This paper addresses this limitation by proposing Operator Deep Q-Learning, a framework that treats the mapping from a reward function to a value function as a learnable operator. This approach enables "zero-shot" reward transfer, allowing agents to generalize to entirely new, unseen reward functions instantly without further training.
2. Methodology
2.1 The Operator View
The core innovation is the use of a Resolvent Operator. Instead of learning a standard Q-value $Q(s, a)$ for a fixed reward, the framework learns an operator $G[r](s, a)$ that takes an entire reward function $r$ as input and outputs the corresponding value function. During the offline training phase, reward functions are sampled from a distribution, and the agent learns to satisfy the Bellman equation across this entire family of rewards simultaneously.
2.2 Theoretically-Informed Architectures
General-purpose operator networks (like DeepONet) often fail in RL because they do not account for the specific mathematical properties of Bellman operators, such as monotonicity and linearity. The authors propose novel designs tailored to these properties:
- Linear/Successor Feature Design: A baseline design where the value is a dot product of state features and reward-dependent weights.
- Attention-based Design (for Policy Evaluation): Since value functions are essentially weighted averages of future rewards, a softmax-based attention mechanism is used to learn weights over a set of "reference points" in the state-action space.
- Max-Out Architecture (for Policy Optimization): To handle the non-linearity of the optimal Bellman equation (due to the
maxoperator), this design maintains multiple linear operator "heads" and takes the maximum over them, mimicking the structure of the optimal policy.
3. Experimental Evaluation
The framework was evaluated in an offline RL setting using three continuous control environments modified for multi-tasking:
- Pendulum-Angle: Swinging a bar to a target angle.
- HalfCheetah-Vel: Running at various target velocities.
- Ant-Dir: Moving a complex 3D quadruped in different directions.
Datasets were constructed from a mix of expert replay buffers and exploratory buffers to test the robustness of the operator learning under varying data quality.
4. Key Results
- Superior Policy Evaluation: The attention-based operator significantly outperformed Vanilla operator networks and Successor Features in Offline Policy Evaluation (OPE), achieving much lower Mean Squared Error (MSE) on unseen rewards.
- Effective Zero-Shot Optimization: The Max-Out Operator achieved zero-shot performance comparable to specialized agents trained online for specific tasks, particularly in the complex Ant-Dir environment.
- Efficiency: The linear decomposition design allows for faster computation ($O(b+m)$ complexity) compared to standard attention, making it practical for large-scale offline datasets.
5. Conclusion
Operator Deep Q-Learning provides a powerful path toward more generalizable RL agents. By learning the underlying relationship between rewards and values, agents can pivot to new tasks without additional environment interaction. While the method currently depends on a reward sampler during training, it sets the stage for future work in reward-free exploration and more robust offline generalization.
Summary updated