TL;DR
Training interpretable reinforcement learning (RL) models, like decision trees, is notoriously slow because evolutionary algorithms require thousands of environment interactions for every candidate solution. Social Interpretable Reinforcement Learning (SIRL) slashes this training time by up to 76% by allowing a population of agents to learn "socially" through a shared collaborative phase before refining their skills individually.
What Problem Does This Paper Address?
While Deep RL has achieved incredible feats, its "black-box" nature makes it risky for high-stakes fields like medicine, law, or autonomous driving. We need Interpretable Reinforcement Learning (IRL)—models like Decision Trees that humans can actually inspect and validate.
The catch? The current state-of-the-art for building these trees often involves Evolutionary Algorithms. In these setups, a population of hundreds of candidate trees must each play out many episodes of a game to see which one is best. This creates a massive computational bottleneck. If you have 500 agents and each needs 10 episodes to get a stable score, that’s 5,000 episodes per generation. SIRL aims to break this linear link between population size and training cost.
Methods
The authors introduce a two-phase training cycle inspired by social learning in animals:
- The Collaborative Phase: Instead of every agent running its own private simulation, the entire population interacts with a single shared instance of the environment. Each agent observes the state and proposes an action. A "voting" mechanism (random selection) picks one action to execute. Crucially, because the underlying algorithm uses off-policy Q-Learning, every agent in the population updates its internal knowledge based on that one shared transition $(s, a, r, s')$, regardless of whether its specific action was chosen.
- The Individual Phase: After the group session, agents move to private simulations for a few episodes to refine their specific policies and provide a clean "fitness score" for the evolutionary optimizer.
The "outer loop" of the system uses Grammatical Evolution (GE) to evolve the structure of the decision trees (the "if-then" logic), while the "inner loop" uses Q-Learning to optimize the values at the leaves of those trees.
Datasets
The method was evaluated on six widely recognized RL benchmarks from the Gymnasium suite:
- Classic Control/Box2D: InvertedPendulum-v2, LunarLander-v2.
- MuJoCo (Robotics): Swimmer-v2, Reacher-v4, Hopper-v3, and Walker2d-v3.
These represent a mix of discrete and continuous action spaces, as well as varying levels of complexity in physics simulation.
Key Insights and Ideas
- Learning from Others' Mistakes: The core "aha" moment is that agents don't need to be "in the driver's seat" to learn. By watching a peer take an action and seeing the result, an agent can update its own policy. This is the power of off-policy learning applied to a population.
- Emergent Consensus: In the beginning, the random voting mechanism ensures high exploration (since agents are uninitialized and propose random actions). As training progresses, agents naturally begin to agree on the best actions, causing the "social" policy to converge toward the optimal one.
- The Efficiency Math: By using $e_c$ collaborative episodes for the whole group and $e_i$ individual episodes per agent, the complexity drops from $O(p \cdot g \cdot e)$ to $O(p \cdot g \cdot e_i + g \cdot e_c)$. When the population $p$ is large, the savings are massive.
Key Results
- Speed & Efficiency: SIRL reduced the number of required training episodes by 43% to 76% across the benchmarks compared to the baseline (ELDT).
- Performance Boost: Despite using fewer episodes, SIRL often achieved higher scores and faster convergence. For example, in the
Walker2d-v3andLunarLander-v2tasks, SIRL significantly outperformed the non-social baseline. - Competitive Interpretability: The resulting models are tiny—often just a few nodes—yet they compete with (and sometimes beat) "opaque" SOTA models like PPO or SAC in terms of raw score, while remaining fully human-readable.
Limitations and Future Work
- Semantic Features Only: Currently, the model requires "meaningful" input variables (like velocity or angle). It cannot yet "build" its own features from raw pixels (like a CNN would), meaning it isn't suitable for pure vision-based RL tasks without a pre-processor.
- Action Space Scaling: While it handles continuous actions via discretization, very large or high-dimensional action spaces still pose a challenge for the Q-learning inner loop.
- Future Directions: The authors suggest exploring more complex voting schemes (e.g., weighting votes by agent performance) and applying this social learning framework to Deep RL hyperparameter optimization.
Summary updated