Introduction
In the fields of materials science and physical chemistry, experimental processes often defy the "Markov property"—the assumption that the current observable state contains all necessary information for optimal decision-making. Researchers from the National Research Council of Canada and the University of Waterloo address this challenge by exploring Reinforcement Learning (RL) within Semi-Markov or Partially Observable Markov Decision Processes (POMDPs). These environments are characterized by "stalling" behaviors, such as latent heat during phase transitions, where energy input does not result in immediate temperature changes, rendering standard RL algorithms ineffective.
The Phase Change Environment
To simulate these real-world complexities, the authors introduced the Phase Change Environment, a benchmark built on the OpenAI Gym framework. In this environment, an agent navigates a 2D grid representing Temperature ($T$) and Pressure ($P$) to reach a specific goal state.
The environment features "phase change boundaries" that require specific sequences of actions to cross, simulating the energy requirements of physical phase transitions. Three difficulty levels were established—EXeasy, EXmod, and EXhard—based on the number of boundaries the agent must navigate.
Methodology
The study evaluates three primary RL approaches:
- Deep Q-Network (DQN): A standard feed-forward network mapping states to actions.
- Deep Recurrent Q-Network (DRQN): Utilizing a Gated Recurrent Unit (GRU), this architecture provides the agent with "memory," allowing it to integrate past observations to infer hidden state information.
- Hindsight Experience Replay (HER): A strategy that allows agents to learn from failed trajectories by treating the final reached state as a pseudo-goal, which is particularly effective in sparse reward environments.
Results and Discussion
The experimental results highlight a significant performance gap between standard and recurrent architectures. While DQN performed adequately in a Markovian control environment, it failed to converge on the "EXhard" task in the Semi-Markov environment, even after 20,000 episodes. The DQN agent's inability to account for "stalled" progress at boundaries led to a performance plateau.
In contrast, DRQN demonstrated the necessity of memory for physical tasks. By building an internal representation of hidden progress during phase changes, DRQN successfully navigated boundaries that DQN could not.
The combination of DRQN + HER emerged as the most robust solution, converging to approximately 145 steps within 4,000 episodes on the hardest task. While this represents a major improvement over DQN, it remains roughly three times higher than the theoretical optimal policy (~40-50 steps), suggesting that recurrent networks do not yet fully capture the complexity of non-equilibrium physical transitions.
Conclusion and Future Work
This research demonstrates that memory-augmented RL is essential for navigating the non-Markovian nature of materials science. Future work will focus on increasing DRQN trace lengths for better long-term memory, incorporating complex spectroscopic data instead of discretized sensors, and addressing non-equilibrium states where the timing of actions is critical. The Phase Change Environment is open-source and intended to serve as a foundation for further RL applications in scientific discovery.
Summary updated