Introduction
Offline Reinforcement Learning (RL) offers the tantalizing promise of learning optimal policies from fixed datasets without requiring costly or dangerous interactions with real-world environments. While this approach has seen success in continuous control tasks (like robotics) or environments with small discrete action sets, it struggles significantly when the action space becomes large and discrete.
In real-world applications like recommender systems with thousands of items or dialogue systems with vast response pools, the "curse" of large action spaces in the offline setting is twofold:
- Generalization Failure: Standard value functions lack semantic understanding. They treat "Action #402" and "Action #403" as completely unrelated entities, even if their effects on the environment are identical.
- Overestimation of Out-of-Distribution (O.O.D.) Actions: Offline RL algorithms typically constrain policies to stay "close to the data" to avoid overestimating the value of unseen actions. In massive action spaces, the vast majority of actions are O.O.D. This forces the agent to either make wildly optimistic mistakes on unseen actions or become overly restrictive, missing out on optimal policies.
To bridge this gap, researchers have introduced MERLION (pseudoMEtRic-based action representations for offLIne reinfOrcemeNt learning), a novel framework that maps actions into a latent space where distances reflect both environmental impact and data frequency.
Methods: The MERLION Framework
MERLION solves the large discrete action problem by learning a state-conditional action representation based on a pseudometric.
The Pseudometric
Instead of relying on arbitrary embeddings, MERLION forces the distance between two action representations to match a specific metric that combines two critical relations:
- Behavioral Relation: This measures the difference in immediate rewards and the resulting next-state distributions (calculated via Wasserstein distance). Simply put, if two actions yield the same outcome in a given state, they should be close together in the latent space.
- Data-distributional Relation: This introduces a penalty term that pushes O.O.D. actions away from in-distribution actions. This structural penalty helps the policy distinguish between "safe" actions present in the dataset and "risky" unseen ones.
Architecture and Training
The architecture relies on an Action Encoder that generates latent representations conditioned on the current state.
Training occurs in two phases:
- Representation Learning: The system samples pairs of actions and uses auxiliary Reward and Transition models to estimate the "behavioral distance" for actions not present in the dataset. The encoder is trained to minimize the difference between the latent distance and this estimated pseudometric.
- Offline RL Execution: A standard offline RL algorithm (such as CQL or BCQ) is trained using these continuous latent representations instead of raw discrete IDs. During execution, the policy outputs a latent vector, which is mapped back to the closest valid discrete action using an $L_1$ nearest-neighbor lookup.
Experimental Setup
The researchers evaluated MERLION across four distinct environments, using datasets generated by sub-optimal online policies with added noise to simulate realistic, imperfect logged data:
- Maze: A grid-world with 4,096 discrete actions (based on 8 actuators).
- Multi-step Maze: A complex version requiring 6-step planning decisions (4,096 actions).
- Video Recommendation (RecSim): A simulator where the agent recommends one of 1,000 videos to users.
- Dialogue System (MultiWOZ): A task-oriented dialogue environment with 1,500 possible dialogue actions.
Key Insights and Results
A core theoretical contribution of MERLION is proving that enforcing this pseudometric makes the Q-value function Lipschitz continuous. This mathematical foundation guarantees that behaviorally similar actions will have similar predicted values, enabling safe generalization.
Furthermore, MERLION captures state-conditional similarity. For example, "move left" and "move right" are distinct actions in an open hallway but functionally identical if an agent is trapped in a corner where both result in a collision. MERLION dynamically adjusts action embeddings based on these state contexts.
Empirically, MERLION demonstrated overwhelming success:
- Superior Performance: MERLION-CQL and MERLION-BCQ significantly outperformed their standard "discrete" counterparts across all four tasks. In many scenarios, discrete baselines failed to even surpass the behavioral policy (the raw data source), whereas MERLION achieved substantial improvements.
- Faster Convergence: The structured latent space allowed the RL agents to discover better policies much faster than training on raw action IDs.
- Ablation Success: Removing the O.O.D. penalty or the distance-learning objective led to noticeable performance degradation, confirming that both the behavioral and distributional components of the metric are essential for success.
Limitations and Future Work
While highly effective, MERLION's framework relies heavily on learning accurate transition and reward models to estimate distances for O.O.D. actions. If these auxiliary models are inaccurate, the resulting action representations will be flawed. Additionally, the nearest-neighbor lookup during execution is $O(N)$. While perfectly efficient for thousands of actions, it could become a computational bottleneck for extreme-scale systems with millions of actions.
Future work will likely explore more sophisticated ways to handle the O.O.D. penalty and integrate the MERLION representation space with a broader variety of offline RL algorithms.
Summary updated