Reinforcement Learning in a Physics-Inspired Semi-Markov Environment

Colin Bellinger, Rory Coles, Mark Crowley, Isaac Tamblyn

Introduction

In the fields of materials science and physical chemistry, experimental processes often defy the "Markov property"—the assumption that the current observable state contains all necessary information for optimal decision-making. Researchers from the National Research Council of Canada and the University of Waterloo address this challenge by exploring Reinforcement Learning (RL) within Semi-Markov or Partially Observable Markov Decision Processes (POMDPs). These environments are characterized by "stalling" behaviors, such as latent heat during phase transitions, where energy input does not result in immediate temperature changes, rendering standard RL algorithms ineffective.

The Phase Change Environment

To simulate these real-world complexities, the authors introduced the Phase Change Environment, a benchmark built on the OpenAI Gym framework. In this environment, an agent navigates a 2D grid representing Temperature ($T$) and Pressure ($P$) to reach a specific goal state.

The Phase Change Environment grid showing Pressure vs. Temperature, including start states of varying difficulty (EXeasy, EXmod, EXhard) and the goal state separated by phase change boundaries.

The environment features "phase change boundaries" that require specific sequences of actions to cross, simulating the energy requirements of physical phase transitions. Three difficulty levels were established—EXeasy, EXmod, and EXhard—based on the number of boundaries the agent must navigate.

Methodology

The study evaluates three primary RL approaches:

  1. Deep Q-Network (DQN): A standard feed-forward network mapping states to actions.
  2. Deep Recurrent Q-Network (DRQN): Utilizing a Gated Recurrent Unit (GRU), this architecture provides the agent with "memory," allowing it to integrate past observations to infer hidden state information.
  3. Hindsight Experience Replay (HER): A strategy that allows agents to learn from failed trajectories by treating the final reached state as a pseudo-goal, which is particularly effective in sparse reward environments.

Results and Discussion

The experimental results highlight a significant performance gap between standard and recurrent architectures. While DQN performed adequately in a Markovian control environment, it failed to converge on the "EXhard" task in the Semi-Markov environment, even after 20,000 episodes. The DQN agent's inability to account for "stalled" progress at boundaries led to a performance plateau.

In contrast, DRQN demonstrated the necessity of memory for physical tasks. By building an internal representation of hidden progress during phase changes, DRQN successfully navigated boundaries that DQN could not.

Performance of DRQN across different difficulty levels, showing successful convergence in steps to goal over training episodes compared to the failure of standard DQN models.

The combination of DRQN + HER emerged as the most robust solution, converging to approximately 145 steps within 4,000 episodes on the hardest task. While this represents a major improvement over DQN, it remains roughly three times higher than the theoretical optimal policy (~40-50 steps), suggesting that recurrent networks do not yet fully capture the complexity of non-equilibrium physical transitions.

Conclusion and Future Work

This research demonstrates that memory-augmented RL is essential for navigating the non-Markovian nature of materials science. Future work will focus on increasing DRQN trace lengths for better long-term memory, incorporating complex spectroscopic data instead of discretized sensors, and addressing non-equilibrium states where the timing of actions is critical. The Phase Change Environment is open-source and intended to serve as a foundation for further RL applications in scientific discovery.

Summary updated

Abstract

Reinforcement learning (RL) has been demonstrated to have great potential in many applications of scientific discovery and design. Recent work includes, for example, the design of new structures and compositions of molecules for therapeutic drugs. Much of the existing work related to the application of RL to scientific domains, however, assumes that the available state representation obeys the Markov property. For reasons associated with time, cost, sensor accuracy, and gaps in scientific knowledge, many scientific design and discovery problems do not satisfy the Markov property. Thus, something other than a Markov decision process (MDP) should be used to plan / find the optimal policy. In this paper, we present a physics-inspired semi-Markov RL environment, namely the phase change environment. In addition, we evaluate the performance of value-based RL algorithms for both MDPs and partially observable MDPs (POMDPs) on the proposed environment. Our results demonstrate deep recurrent Q-networks (DRQN) significantly outperform deep Q-networks (DQN), and that DRQNs benefit from training with hindsight experience replay. Implications for the use of semi-Markovian RL and POMDPs for scientific laboratories are also discussed.
BibTeX
@inproceedings{bellinger2020reinforcement,
  author = {Colin Bellinger and Rory Coles and Mark Crowley and Isaac Tamblyn},
  title = {{Reinforcement Learning in a Physics-Inspired Semi-Markov Environment}},
  booktitle = {Canadian AI},
  year = {2020},
  doi = {10.1007/978-3-030-47358-7\_6},
}
Figure