AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-Aware Prioritization

Kailash Gogineni, Yongsheng Mei, Tian Lan, Peng Wei, Guru Venkataramani

Abstract

Multi-Agent Reinforcement Learning (MARL) faces significant scalability challenges due to the massive data requirements of centralized training. Current algorithms often suffer from memory bottlenecks caused by "cache-unfriendly" random access patterns during experience replay. This paper introduces AccMER (Accelerated Multi-Agent Experience Replay), a framework that optimizes hardware utilization by intelligently reusing high-priority data across multiple training steps. By improving cache locality and reducing data movement between RAM and the processor, AccMER achieves a 25.4% reduction in training time while maintaining the performance and convergence of state-of-the-art MARL algorithms.

Introduction

In the Centralized Training and Decentralized Execution (CTDE) paradigm, agents learn from a shared replay buffer containing the collective experiences of the entire swarm. As the number of agents increases, the volume of data shuffled between main memory (RAM) and the processor's local cache grows quadratically. Standard experience replay methods typically sample this data uniformly or based on priority, resulting in stochastic access patterns that frequently miss the fast local cache. These "cache misses" force the processor to wait for slow retrievals from RAM, creating a bottleneck that can account for the vast majority of total training time.

The Memory Bottleneck in MARL

Profiling of large-scale MARL systems reveals that the "Update" phase—where trainers sample mini-batches and update neural network weights—is the primary performance sink. For a system with 48 agents, this phase can consume up to 86% of total training time. Within that update cycle, the simple act of sampling and moving mini-batches accounts for 60% of the overhead. The fundamental conflict lies between the mathematical need for randomness (to ensure stable learning) and the hardware need for predictable data patterns (to ensure cache locality).

The AccMER Framework

AccMER addresses this bottleneck through a hardware-software co-design approach that prioritizes data locality without sacrificing algorithmic integrity. The framework consists of four key components:

  1. Cache-Aware Data Reuse: Rather than fetching a completely new random batch at every training step, AccMER identifies a subset of high-priority transitions and retains them in the cache for a window of $n$ steps. This significantly reduces the frequency of expensive RAM-to-cache transfers.
  2. Optimal Prioritization: To select which data is most "worthy" of cache retention, AccMER utilizes a prioritization scheme based on the Bellman error (the discrepancy between predicted and actual rewards) and the joint action probability (the likelihood of specific coordinated agent actions).
  3. Hybrid Sampling Strategy: To prevent the model from overfitting on reused data, AccMER employs a split-batch approach. It combines a reused set ($S^-$), containing high-priority transitions kept in the cache, with a fresh set ($S^+$), consisting of new transitions sampled uniformly to maintain data diversity.
  4. Weight Discounting: A discount factor is applied to the weights of reused transitions. This ensures that the training process remains balanced and does not become overly biased toward the specific experiences held in the cache.

Experimental Evaluation

The researchers evaluated AccMER using a difficulty-enhanced version of the Predator-Prey environment, featuring 8 to 32 agents on a 10x10 grid. This setup required high levels of coordination, as agents were punished for failed capture attempts. The study compared AccMER against established benchmarks, including QMIX, WQMIX, and the state-of-the-art MAC-PO algorithm.

Key Results

  • Training Speedup: AccMER reduced end-to-end training time by 25.4% for 32-agent scenarios compared to MAC-PO.
  • Hardware Efficiency: The framework successfully optimized memory access, reducing Last-Level Cache (LLC) load misses by 23.7% and global cache misses by 17.2%.
  • Algorithmic Parity: Despite the reuse of data, AccMER maintained mean rewards and convergence speeds equivalent to (and occasionally better than) top-tier MARL algorithms, proving that hardware optimization does not have to come at the cost of agent intelligence.

Conclusion and Future Work

AccMER demonstrates that significant performance gains in MARL can be achieved by looking beyond algorithmic logic to the underlying hardware execution. By aligning data sampling patterns with the architectural realities of modern processors, AccMER mitigates the quadratic scaling costs of multi-agent systems.

Future research directions include testing the framework on more complex environments like the StarCraft Multi-Agent Challenge (SMAC) and developing methods to dynamically adjust the "reuse ratio" based on real-time hardware telemetry. Additionally, combining AccMER with other acceleration techniques, such as quantization, could further push the boundaries of large-scale agent training.

Summary updated

BibTeX
@inproceedings{gogineni2023accmer,
  author = {Kailash Gogineni and Yongsheng Mei and Tian Lan and Peng Wei and Guru Venkataramani},
  title = {{AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-Aware Prioritization}},
  booktitle = {ASAP},
  year = {2023},
  doi = {10.1109/asap57973.2023.00041},
}
Figure