AMRL: Aggregated Memory For Reinforcement Learning

Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, Katja Hofmann

Standard memory models like Long Short-Term Memory (LSTM) networks often struggle in Reinforcement Learning (RL) because they are inherently designed to prioritize recent events and specific sequences. To address this, researchers have introduced AMRL (Aggregated Memory for Reinforcement Learning), an architecture that augments LSTMs with order-invariant aggregators. This hybrid approach maintains a high signal-to-noise ratio over long horizons, leading to significantly better performance in noisy, complex environments.

The Problem: Partial Observability and Recency Bias

In many RL tasks, an agent only sees a small part of its world at any given time—a condition known as partial observability. To succeed, the agent must rely on memory to store past observations, such as the location of a key or a passcode seen earlier in an episode.

The industry standard has been to borrow architectures from Natural Language Processing (NLP), specifically LSTMs. However, LSTMs exhibit two major flaws when applied to RL:

  1. Recency Bias: LSTMs are designed for tasks where the most recent inputs are typically the most important. In RL, an observation from 1,000 steps ago might be the critical piece of information needed right now.
  2. Sensitivity to Noise: RL environments are inherently noisy. Between seeing a "passcode" and reaching a "door," an agent might wander aimlessly or process thousands of irrelevant visual frames. In a standard LSTM, this "exploration noise" effectively washes out the important historical signal.

Methods: The AMRL Architecture

To overcome these limitations, the authors propose AMRL, a hybrid architecture that combines the strengths of sequential memory with the robustness of set-based aggregation.

AMRL Architecture Comparison
  • The Hybrid Approach: As illustrated above, the model uses a standard LSTM to capture short-term context and order-dependent information. Simultaneously, it takes a portion of the LSTM's features and feeds them into an Aggregator module.
  • Order-Invariant Aggregators: The researchers tested three types of aggregators: SUM, AVG, and MAX. Unlike LSTMs, these functions treat all past observations as a "bag of facts." For instance, if an agent saw a red key at step 5, the MAX aggregator will remember that "redness" just as clearly at step 500, regardless of the sequence of events in between.
  • Straight-Through (ST) Connections: To ensure the model can learn from events far in the past, AMRL utilizes "straight-through" gradients. This prevents the mathematical signal from vanishing as it travels back through time during training (Backpropagation Through Time).

Key Insights and Results

The researchers evaluated AMRL across domains designed to stress-test long-term memory, including symbolic TMaze tasks and high-dimensional 3D Minecraft environments (via the Malmo platform).

Order-Invariance as a Feature: While NLP requires strict order (e.g., "The dog bit the man" vs. "The man bit the dog"), many RL tasks just need to know that something happened, not exactly when. By ignoring order for long-term facts, AMRL becomes immune to the noise of the agent's own exploration path.

Maintaining Signal-to-Noise Ratio (SNR): Formal analysis shows that an LSTM's SNR decays exponentially over time. In contrast, AMRL’s SNR decays only linearly (for AVG/SUM) or remains constant (for MAX), allowing the agent to "see" important past events through the fog of current noise.

AMRL vs LSTM Performance

Superior Performance and Efficiency: In the hardest tasks, AMRL-Max and AMRL-Avg reached near-optimal returns while standard LSTMs and more complex models like the Differentiable Neural Computer (DNC) struggled or failed entirely. AMRL improved average return by 19% over an LSTM baseline with the same number of parameters, and outperformed a much larger DNC baseline by 9%.

Crucially, when observation noise was introduced, the LSTM's performance plummeted immediately, whereas AMRL maintained near-identical learning speeds, proving its robust resilience to noise.

Limitations and Future Work

While highly effective, AMRL has a few limitations:

  • The "Sum" Problem: The SUM aggregator can sometimes grow too large over very long episodes, making it difficult for the network to interpret the magnitude of the signal.
  • Interference: There is a risk that short-term tasks could interfere with long-term goals. Future work could explore better gating mechanisms to control what enters the aggregated memory.
  • Complex Dependencies: If a task requires remembering a very specific sequence of events over a long period (e.g., "First A, then B, then C"), the order-invariant nature of the aggregator might be a disadvantage, though the standard LSTM component is designed to help mitigate this.

Summary updated

BibTeX
@inproceedings{beck2020amrl,
  author = {Jacob Beck and Kamil Ciosek and Sam Devlin and Sebastian Tschiatschek and Cheng Zhang and Katja Hofmann},
  title = {{AMRL: Aggregated Memory For Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure