Meta-learning how to Share Credit among Macro-Actions

Ionel-Alexandru Hosu, Traian Rebedea, Razvan Pascanu

TL;DR

Adding "macro-actions" (sequences of primitive actions) to reinforcement learning agents is supposed to simplify long-horizon exploration, but it often backfires by bloating the action space. This paper introduces the Macro-Action Similarity Penalty (MASP), a method that meta-learns a similarity matrix to share credit between related actions. By exploiting the underlying geometry of the action space, MASP prevents performance degradation and significantly outperforms strong baselines on Atari and StreetFighter II.

What Problem Does This Paper Address?

In reinforcement learning (RL), sparse rewards and long episodes make credit assignment—figuring out which specific action led to a reward hours later—incredibly difficult. One classic remedy is using macro-actions: pre-defined sequences of primitive actions (like "dash-forward-punch" in a fighting game). In theory, macro-actions make the episode shorter because the agent makes fewer high-level decisions.

In practice, however, naively adding macro-actions often hurts performance. Why? Because standard exploration strategies (like $\epsilon$-greedy) treat every action as completely independent and atomic. If you add 100 macro-actions to an environment with 10 primitive actions, you have drastically increased the branching factor of your search space. The authors argue that the key to fixing this paradox is recognizing that macro-actions are inherently related to each other and to the primitive actions they are built from.

Methods

The authors propose Macro-Action Similarity Penalty (MASP), a novel regularization framework that enforces structure on the action space.

  1. The MASP Loss: The core idea is a "soft" penalty added to the standard Temporal Difference (TD) loss. It penalizes large differences between the Q-values of actions that are functionally similar: $$\mathcal{L}_{\text{MASP}} = \eta \cdot \|Q(s, \cdot; \theta) - \Sigma Q(s, \cdot; \theta)\|_2^2$$ Where $\Sigma$ is a similarity matrix and $\eta$ is a regularization coefficient. If two actions are highly similar according to $\Sigma$, the network is forced to update their Q-values together.

  2. Meta-Learning the Similarity Matrix ($\Sigma$): Rather than hand-crafting $\Sigma$, the authors use a meta-gradient framework. Training alternates between two phases:

  3. Inner Loop: The agent updates its policy network parameters ($\theta$) using the combined TD and MASP loss while keeping $\Sigma$ fixed.
  4. Outer Loop: The updated policy is evaluated on a held-out trajectory. The gradient of this meta-objective is backpropagated to update $\Sigma$, dynamically learning which actions behave similarly.
  5. Handling Non-Stationarity: Because $\Sigma$ changes during training, the value function could struggle to adapt. To fix this, the authors flatten $\Sigma$, project it into a low-dimensional embedding ($e_\Sigma$), and feed it directly into the Q-network as additional context (similar to a Universal Value Function Approximator).

Datasets

The method was rigorously evaluated on three distinct setups:

  • Atari 2600 Games (ALE): Evaluated across a wide suite of classic games. Macro-actions were extracted by finding the most frequent action subsequences (lengths 3 to 8) within human expert trajectories from the Atari Grand Challenge Dataset.
  • StreetFighter II (Gym Retro): A complex environment where macro-actions represent actual in-game special move combos.
  • MiniGrid: Procedurally-generated gridworlds requiring structured navigation and planning.

Key Insights and Ideas

  • Action Space Geometry Matters: Treating macro-actions as isolated choices ignores the fact that they share primitive components. MASP introduces an inductive bias that allows the agent to learn about entire clusters of actions simultaneously.
  • Soft Regularization vs. Hard Hierarchies: Unlike traditional hierarchical RL (like the Options framework), which forces rigid sub-policy choices, MASP's soft penalty allows the agent to naturally group actions early in training and fine-tune the subtle differences between them later.
  • Clustering the Noise: When the action space is flooded with useless or corrupted macro-actions, MASP automatically learns to cluster those non-informative actions together and ignore them, preventing the policy collapse that plagues baseline algorithms.

Key Results

The authors evaluated MASP against standard Rainbow-DQN and a baseline augmented with naive macro-actions (Rainbow-DQN + Macro-Actions).

Visual comparison of mean scores across Atari and Street Fighter II
  • Superior Performance: MASP consistently outperformed both baselines. In many Atari games, it achieved scores well above human-level benchmarks where the naive macro-action baseline failed to even match standard Rainbow-DQN.
  • Robustness to Action Bloat: In an ablation study varying the number of macro-actions ($k \in [64, 1024]$), the naive baseline's performance collapsed catastrophically as $k$ increased. MASP maintained stable, strong performance even with 1,024 actions.
  • Robustness to Noise: When macro-actions were intentionally corrupted (replaced with random sequences with up to 75% probability), MASP remained highly resilient, while the naive macro baseline degraded sharply.
  • Transfer Learning: Crucially, the authors demonstrated that a similarity matrix $\Sigma$ meta-learned on one game (e.g., Montezuma's Revenge) could be frozen and successfully transferred to a structurally similar game (e.g., Private Eye), accelerating learning without retraining $\Sigma$ from scratch.

Limitations and Future Work

  • Quadratic Computational Cost: The similarity matrix $\Sigma$ scales quadratically ($|A| \times |A|$) with the size of the action space. For extremely large or continuous action spaces, maintaining and projecting this matrix becomes computationally prohibitive.
  • State-Independent Similarity: Currently, $\Sigma$ is global and independent of the environment's current state. Incorporating state-dependence could capture richer action dynamics (e.g., a macro-action might only be similar to another when facing a wall) but would make meta-learning significantly harder.
  • Dependence on Initial Proposals: MASP manages an expanded action space beautifully, but it still relies on an external heuristic (like human datasets or domain knowledge) to propose the macro-actions in the first place. Integrating automatic online macro-discovery remains an open challenge.

Summary updated

Abstract

One proposed mechanism to improve exploration in reinforcement learning is through the use of macro-actions. Paradoxically though, in many scenarios the naive addition of macro-actions does not lead to better exploration, but rather the opposite. It has been argued that this was caused by adding non-useful macros and multiple works have focused on mechanisms to discover effectively environment-specific useful macros. In this work, we take a slightly different perspective. We argue that the difficulty stems from the trade-offs between reducing the average number of decisions per episode versus increasing the size of the action space. Namely, one typically treats each potential macro-action as independent and atomic, hence strictly increasing the search space and making typical exploration strategies inefficient. To address this problem we propose a novel regularization term that exploits the relationship between actions and macro-actions to improve the credit assignment mechanism by reducing the effective dimension of the action space and, therefore, improving exploration. The term relies on a similarity matrix that is meta-learned jointly with learning the desired policy. We empirically validate our strategy looking at macro-actions in Atari games, and the StreetFighter II environment. Our results show significant improvements over the Rainbow-DQN baseline in all environments. Additionally, we show that the macro-action similarity is transferable to related environments. We believe this work is a small but important step towards understanding how the similarity-imposed geometry on the action space can be exploited to improve credit assignment and exploration, therefore making learning more effective.
BibTeX
@inproceedings{hosu2025meta,
  author = {Ionel-Alexandru Hosu and Traian Rebedea and Razvan Pascanu},
  title = {{Meta-learning how to Share Credit among Macro-Actions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=cJlgdpEFx9},
}
Figure