Policy Gradient RL Algorithms as Directed Acyclic Graphs

Juan Jose Garau-Luis, Juan Jose Garau LuisarXiv 2020 · arXiv

Abstract

This paper expands the vocabulary of automated reinforcement learning (RL) by representing complex Policy Gradient (PG) algorithms as Directed Acyclic Graphs (DAGs). By introducing new mathematical primitives to an existing meta-learning framework, we successfully encode five foundational RL algorithms—VPG, PPO, DDPG, TD3, and SAC. This structural contribution demonstrates the modularity of modern RL methods and provides a foundation for evolving more sophisticated learning rules for continuous control tasks.

Introduction

Designing new reinforcement learning (RL) algorithms is traditionally a labor-intensive, human-driven process. To scale this, researchers have turned to Meta-RL, where a meta-learner automatically discovers optimal "inner" update rules for an agent. A recent breakthrough represented these rules as Directed Acyclic Graphs (DAGs), where nodes represent mathematical operations and edges represent data flow. While effective for "rediscovering" algorithms like DQN, this approach previously lacked the expressive power for Policy Gradient (PG) methods—the industry standard for continuous action spaces. This work closes that gap by expanding the search language to accommodate the specific operations required by PG methods, such as discounted advantage estimation and probability ratio clipping.

Methods and Search Language Extension

The core of this work involves extending the evolutionary meta-learning framework introduced by Co-Reyes et al. (2021). The "search language"—the set of available nodes in the DAG—was expanded to include several key primitives:

The evolutionary meta-learning framework for discovering RL algorithms represented as DAGs.
  • New Input Nodes: Signals for episode termination ($d_t$), the $\lambda$ parameter for advantage estimation, and support for sequence processing.
  • New Operation Nodes: SumAndDiscount for calculating rewards-to-go or Generalized Advantage Estimation (GAE); Clip for the stability logic found in PPO; and Squashing & Prob for handling continuous action spaces in algorithms like Soft Actor-Critic (SAC).
  • Parameter Mapping: New nodes allow neural networks to map both states and actions to a single value ($NN: S, Z \to \mathbb{R}$), enabling the representation of Q-functions in actor-critic architectures.

Encoding Foundational Algorithms

The primary result of this work is the successful manual construction of DAGs for five major RL algorithms. This "proof of concept" demonstrates that the extended search language is sufficiently complete to represent modern RL logic.

  • VPG: Represented through separate loss functions for policy and value.
  • PPO: Successfully incorporates complex clipping logic and probability ratios.
  • DDPG/TD3: Handles continuous action spaces and "Twin" Q-network architectures.
  • SAC: Integrates entropy hyperparameters and squashed Gaussian policies.
DAG representation of the PPO algorithm, incorporating clipping logic and probability ratios.

Key Insights

Representing algorithms as DAGs offers significant advantages in interpretability. Unlike black-box meta-learned rules, DAGs provide a readable flowchart of the underlying mathematical logic. Furthermore, this work highlights the modularity of RL; complex algorithms like SAC are shown to be compositions of simpler primitives. Finally, these human-designed DAGs can serve as "warm starts" for evolutionary search, allowing the meta-learner to mutate proven designs rather than starting from scratch.

Limitations and Future Work

This paper focuses on the structural representation and language extension. While it defines the graphs, it does not yet report results from a full evolutionary search using these new nodes to discover entirely novel algorithms. Future work will focus on empirical validation—using these PG graphs as seeds to evolve specialized versions like "Super-PPO"—and expanding the DAG library to include Model-Based and Multi-Agent RL.

Summary updated

BibTeX
@misc{garauluis2020policy,
  author = {Juan Jose Garau-Luis and Juan Jose Garau Luis},
  title = {{Policy Gradient RL Algorithms as Directed Acyclic Graphs}},
  year = {2020},
  eprint = {2012.07763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.07763},
}
Figure