In standard Reinforcement Learning (RL), an agent receives a single scalar reward at each step and attempts to maximize its cumulative sum. However, real-world scenarios are rarely this straightforward. Consider a home entertainment system serving multiple users: if User A provides a reward of +100 and User B provides +1, a standard RL agent will entirely ignore User B.
This paper addresses the limitations of the "single scalar reward" paradigm by treating multiple reward sources as independent players in a game. By applying a Nash Equilibrium framework to a "policy voting" system, the authors develop agents capable of balancing incomparable goals and adapting instantly when reward sources enter or leave the environment.
The Problem with Scalar Rewards
The authors identify three primary issues with collapsing multiple rewards into a single sum:
- Incomparability: Different users or goals operate on different "scales" of reward, which shouldn't inherently dictate their level of importance.
- Dynamic Environments: If a reward source (e.g., a specific user) leaves the environment, a policy trained on a fixed sum of rewards instantly becomes obsolete.
- Strategic Behavior: If users are aware they are training an agent, they might "game" the system by providing extreme reward values to seize control.
To resolve this, the authors propose a method to balance these sources without losing the underlying information about who or what provided the reward.
The Policy Votes Framework
Instead of the agent learning one fixed policy based on summed rewards, the paper introduces a Policy Votes framework. The final policy is an average of "votes" cast by each present reward source.
- The Voting Mechanism: Each source $s$ provides an "effort" $\alpha_s(x)$ (indicating how much it cares about state $x$) and a "vote" $v_s(x, a)$ (indicating its preferred action). The agent's final policy is the weighted average of these votes.
- Game Theory (Nash Equilibrium): The problem is framed as a general-sum game where each reward source acts as a player. The system seeks a Nash Equilibrium: a state where no single reward source can improve its return by changing its votes, given the votes of the others.
- Fictitious Play Algorithm: Because a perfect world model cannot be assumed, the authors use a "fictitious play" approach. They estimate the expected return for each source using a parametric form based on the KL-divergence between a source's "ideal" policy and the agent's actual policy. Votes are iteratively updated using gradient descent until the system converges to an equilibrium.
Experimental Validation
The authors validate their approach using two simulated environments designed to test the agent's ability to handle competing interests and physical bottlenecks.
1. Multiple Delivery Load-Unload Problem
In this environment, an agent must pick up cargo and deliver it to one of several "boxed" states, each representing a different reward source. This tests the agent's ability to distribute service fairly among competing users.
The algorithm successfully found a stochastic policy that balanced deliveries to all users. A key breakthrough here is Zero-Shot Adaptation. Because the agent learns the individual preferences of each source independently, if a user leaves the environment, the agent doesn't need to relearn. It simply recalculates the Nash Equilibrium with the remaining "players," resulting in an immediate, optimal policy shift. Furthermore, by constraining each source to a total "vote" sum of 1, the system naturally prevents one user from dominating simply by using larger reward magnitudes.
2. One-way Door Problem
This is a 15-state navigation task with stochastic transitions where two sources have conflicting goals (one wants the agent at the far left, the other at the far right).
This environment tests how the agent handles physical bottlenecks and conflicting preferences. The agent learned a "natural" solution: it moves to a central conflict state and then chooses a direction. The authors demonstrate that a traditional "sum of rewards" approach would fail here; the agent would likely get stuck circling one loop forever, never discovering the other source's goal. The voting approach forces the agent to explore trade-offs that a simple sum would ignore.
Limitations and Future Work
While the "fictitious play" algorithm typically converged in just a few dozen iterations, the paper notes several limitations:
- Parametric Assumptions: The current model assumes the return is linear in the KL-divergence from a target policy. This works well for "unimodal" preferences (one clear best way to achieve a goal) but struggles with "bimodal" scenarios (e.g., if a goal can be satisfied in two completely different ways).
- Complexity of Preferences: Handling more sophisticated tasks will require more complex models for how a source's preference relates to the policy.
- Scalability: While effective for small state spaces, finding Nash Equilibria in very high-dimensional state and action spaces remains a significant computational challenge for future research.
Summary updated