Abstract
Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.
Introduction
Training Deep Reinforcement Learning (RL) agents in massive, complex environments is often prohibitively slow. In domains like city-wide traffic control or massive warehouse robotics, running a "Global Simulator" (GS) that tracks every single entity is computationally expensive. While using a "Local Simulator" (LS)—simulating just a small portion of the environment—is faster, it typically ignores external influences (e.g., traffic spillover from distant intersections). This creates a distribution shift where agents fail when deployed back into the interconnected global system.
This paper introduces Influence-Augmented Local Simulators (IALS), a method that replaces slow global simulations with lightweight local ones paired with a learned "influence predictor" that mimics the impact of the outside world. The result is a training process that is roughly three times faster while achieving the same performance as training on the full, global system.
Methodology: Influence-Augmented Local Simulators
The IALS architecture consists of two primary components: a fast, stripped-down Local Simulator (LS) that models the agent's immediate vicinity, and an Approximate Influence Predictor (AIP). The AIP is a neural network, often a Recurrent Neural Network (RNN), trained to predict "influence sources"—external variables that directly affect the local area.
Workflow
The training process follows three main steps:
- Data Collection: A slow Global Simulator is run using a random policy to collect a dataset of external influences on the local region.
- AIP Training: The AIP is trained to predict these influences based on local history.
- Fast RL Training: The agent is trained using the IALS. The LS handles local physics, while the AIP provides the "inflow" of external events.
A key technical contribution is the use of d-separating sets (d-sets). By selecting only the local variables mathematically necessary to predict external influence, the authors prevent the AIP from picking up spurious correlations, ensuring the simulator remains robust as the agent's policy evolves.
Experimental Evaluation
The approach was evaluated on two networked systems:
- Traffic Control: A grid of 25 intersections (using Flow and SUMO). The agent controls one intersection while others are scripted.
- Warehouse Commissioning: A warehouse with 36 robots. The agent controls one robot while 35 others compete for items.
Key Insights and Results
The experiments yielded several critical findings:
- 3x Training Speedup: IALS reached peak performance in approximately one-third of the wall-clock time required by the Global Simulator.
- Performance Parity: Agents trained on the approximate IALS achieved the same final reward as those trained on the full global model.
- Memory Matching: Theorem 1 demonstrates that the influence predictor only needs as much memory as the agent itself to be effective. If the agent looks at a 10-second window, the predictor only needs to model the world's impact over that same window.
- Robustness via d-sets: Using a minimal "d-set" prevents feedback loops that can break simulations when an agent's behavior changes significantly.
Conversely, "naive" local simulators that ignored external influence performed significantly worse in global testing, highlighting that modeling influence is essential for transferability.
Conclusion and Future Work
IALS offers a scalable path for applying deep RL to large-scale systems. While it currently requires an initial "cold start" data collection phase from a global simulator, it drastically reduces the main training bottleneck. Future work includes addressing policy-dependent influence in highly reactive systems and scaling the framework to multi-agent training scenarios where hundreds of agents might be trained simultaneously using local influence models.
Summary updated