Reinforcement Learning with Temporal-Logic-Based Causal Diagrams

Yash Paliwal, Rajarshi Roy, Jean-Raphaël Gaglione, Nasim Baharisangari, Daniel Neider, Xiaoming Duan, Ufuk Topcu, Zhe Xu

Abstract

Standard Reinforcement Learning (RL) agents often struggle with sample efficiency because they treat the environment as a black box, learning complex temporal goals solely through trial and error. This paper introduces Temporal-Logic-based Causal Diagrams (TL-CDs), a framework that integrates causal knowledge into the RL process using symbolic logic. By formalizing environmental laws through Linear Temporal Logic (LTLf), the proposed QTLCD algorithm enables agents to identify when a goal has become guaranteed or impossible before an episode concludes. Experimental results across various grid-world environments demonstrate that this approach significantly reduces redundant exploration and accelerates convergence to optimal policies.

Introduction

In complex environments, agents are often tasked with achieving goals that require specific sequences of actions—such as "collect a key before opening a door while avoiding hazards." While recent research has utilized Deterministic Finite Automata (DFA) to represent these reward functions, these models typically only describe what the agent should do, not how the environment's causal laws constrain those actions.

Without causal awareness, an RL agent may continue to explore a state space long after it has entered a "dead-end" where the goal is no longer achievable. Conversely, it may continue taking steps in an episode where the goal is already effectively guaranteed. This lack of foresight leads to significant computational waste. This paper addresses this gap by bridging Causal Inference and Formal Methods to provide agents with a structured understanding of environmental dynamics.

Methodology

Temporal-Logic-based Causal Diagrams (TL-CD)

Traditional causal diagrams are static, representing relationships between variables at a single point in time. To capture the dynamic nature of RL environments, the authors propose TL-CDs. These diagrams use Linear Temporal Logic (LTLf) to express causal relationships over time. For instance, a TL-CD can formally encode a rule such as: "If a specific resource is consumed now, the agent is permanently barred from reaching a future state."

The QTLCD Algorithm

The core contribution is the Q-learning with TL-CD (QTLCD) algorithm, which extends "Q-learning for Reward Machines" (QRM). The algorithm synchronizes the agent’s progress with two distinct automata:

  1. Task DFA: Represents the specific goal or reward function.
  2. Causal DFA: Represents the underlying rules and constraints of the environment.

By monitoring both, the algorithm implements an Early Stopping mechanism. It identifies two critical types of states:

  • Causally Accepting States: Configurations where the causal laws guarantee the agent will reach the goal regardless of subsequent actions.
  • Causally Rejecting States: "Dead-ends" where the goal has become mathematically impossible to achieve.

When the agent enters either state, the episode terminates immediately. The Q-values are then updated based on the predicted outcome, preventing the agent from wasting steps on determined results.

Experimental Evaluation

The authors evaluated the QTLCD framework in three distinct grid-world domains:

  • Small Office World: A 17x9 grid requiring navigation through one-way doors and sequential key collection.
  • Large Office World: A 25x25 grid designed to test the scalability of the causal framework in larger state spaces.
  • Crossroad Domain: A stochastic environment where an agent must interact with a signal button and wait for a light change governed by a geometric distribution.

Key Insights and Results

Efficiency and Convergence

The inclusion of causal diagrams led to dramatic improvements in training efficiency. In the Small Office World, the QTLCD agent reached an optimal policy approximately 1.5 times faster than the baseline QRM agent. This advantage scaled with the complexity of the environment; in the Large Office World, the QTLCD agent converged nearly 3 times faster than the baseline.

Reduced Exploration

Qualitative analysis indicates that the agent learns to avoid entire branches of the state space that the causal diagram identifies as "causally rejecting." This "Neuro-Symbolic" synergy—using symbolic logic to provide a backbone for numerical learning—allows the agent to effectively "see into the future" and avoid the trial-and-error costs associated with environmental dead-ends.

Limitations and Future Work

While promising, the current framework relies on a ground-truth TL-CD provided by human experts. In many real-world applications, these causal laws may be unknown or partially observable. Future research directions include:

  • Learning Causal Models: Developing methods to extract TL-CDs directly from environmental data.
  • Continuous State Spaces: Adapting the logic-based framework for high-dimensional, continuous control tasks like robotics.
  • Computational Overhead: Addressing the potential state-space explosion that occurs when calculating the cross-product of multiple complex DFAs.

Conclusion

By embedding causal knowledge into the reinforcement learning loop via temporal logic, this research provides a robust method for improving sample efficiency. The ability to terminate episodes early based on causal certainty allows agents to focus their exploration on meaningful states, paving the way for more practical RL applications in complex, rule-bound environments.

Summary updated

Abstract

We study a class of reinforcement learning (RL) tasks where the objective of the agent is to accomplish temporally extended goals. In this setting, a common approach is to represent the tasks as deterministic finite automata (DFA) and integrate them into the state-space for RL algorithms. However, while these machines model the reward function, they often overlook the causal knowledge about the environment. To address this limitation, we propose the Temporal-Logic-based Causal Diagram (TL-CD) in RL, which captures the temporal causal relationships between different properties of the environment. We exploit the TL-CD to devise an RL algorithm in which an agent requires significantly less exploration of the environment. To this end, based on a TL-CD and a task DFA, we identify configurations where the agent can determine the expected rewards early during an exploration. Through a series of case studies, we demonstrate the benefits of using TL-CDs, particularly the faster convergence of the algorithm to an optimal policy due to reduced exploration of the environment.
BibTeX
@inproceedings{paliwal2023reinforcement,
  author = {Yash Paliwal and Rajarshi Roy and Jean-Raphaël Gaglione and Nasim Baharisangari and Daniel Neider and Xiaoming Duan and Ufuk Topcu and Zhe Xu},
  title = {{Reinforcement Learning with Temporal-Logic-Based Causal Diagrams}},
  booktitle = {CD-MAKE},
  year = {2023},
  url = {https://openreview.net/forum?id=akvFEoK7PE},
}
Figure