Q-Cogni: An Integrated Causal Reinforcement Learning Framework

Cris Cunha, Wei Liu, Tim French, Ajmal Mian

Introduction

Reinforcement learning (RL) has traditionally been divided into model-free approaches, which learn habits through trial and error, and model-based approaches, which plan using environmental maps. While model-free agents are computationally efficient, they often suffer from extreme sample inefficiency and lack interpretability, operating as "black boxes." A fundamental limitation in current RL is the lack of causal understanding; agents may correlate actions with rewards without grasping the underlying "why."

Q-Cogni addresses this gap by bridging the divide between habit-based learning and causal reasoning. It is a hybrid RL framework that autonomously discovers the causal structure of an environment to guide the learning process. By integrating a Bayesian Network into a redesigned Q-Learning algorithm, Q-Cogni achieves superior sample efficiency, scalability for high-dimensional routing, and human-readable decision-making.

Methodology

The Q-Cogni framework is composed of three primary modules designed to mimic the human brain's dual-system approach to learning and reasoning.

The Q-Cogni framework architecture, integrating causal discovery, inference, and modified Q-learning.

1. Autonomous Causal Structure Discovery

The process begins with an agent performing a random walk to gather initial environmental data. Using the NOTEARS algorithm, the module identifies a Directed Acyclic Graph (DAG) that represents the causal relationships between states, actions, and rewards. This step transforms raw observational data into a structured causal model.

2. Causal Inference Module

The discovered DAG is converted into a Bayesian Network. This allows the agent to perform causal queries, calculating the probability of achieving specific sub-goals based on current states and potential actions. For example, the agent can determine the likelihood that a specific movement will lead to a successful passenger pickup.

A discovered Directed Acyclic Graph (DAG) for the Taxi-v3 environment, illustrating causal dependencies between states and actions.

3. Modified Q-Learning

The core of the framework is a redesigned Q-Learning algorithm. Instead of relying solely on standard exploration strategies like $\epsilon$-greedy, Q-Cogni uses the Bayesian Network to "shortcut" exploration. If the causal model identifies an action as a direct "parent" of a goal, the agent prioritizes it. Additionally, the Q-function incorporates probabilistic reward adjustments to facilitate learning in environments with sparse rewards.

Experimental Setup

The framework was validated across three distinct environments:

  • Taxi-v3 (OpenAI Gym): A benchmark $5 \times 5$ grid task involving passenger pickup and delivery.
  • Synthetic Large-Scale Grids: Scalability tests ranging from $8 \times 8$ to $512 \times 512$ grids.
  • Real-World NYC TLC Data: Using the OSMNX library, the authors modeled the Astoria region of New York City as a graph with 666 nodes and 1,712 edges to test navigation in complex, real-world topologies.

Results and Discussion

Experimental results demonstrate that Q-Cogni significantly outperforms standard algorithms like PPO and DDQN in terms of both efficiency and stability. In the Taxi-v3 environment, the agent reached optimal rewards faster and with lower variance.

A key finding is the framework's scalability. While traditional shortest-path algorithms like Dijkstra and A* become computationally expensive as node counts increase, Q-Cogni scales near-linearly. In the NYC dataset, Q-Cogni's policies matched or exceeded traditional search methods in 85% of cases.

Furthermore, the framework offers high causal interpretability. Unlike black-box models, Q-Cogni can provide conditional probabilities for its actions, explaining why a specific move was chosen based on its predicted causal impact on the goal. Finally, the causal structure discovered in small environments proved to be domain-agnostic, allowing it to be transferred to larger, real-world maps without retraining the discovery module.

Conclusion and Future Work

Q-Cogni represents a significant step toward integrating causal reasoning into reinforcement learning. By leveraging discovered causal structures, it overcomes traditional limitations in sample efficiency and interpretability. While currently optimized for discrete state-action spaces, future research will focus on extending this framework to continuous variables, which are essential for broader applications in robotics and autonomous control.

Summary updated

BibTeX
@article{cunha2024q,
  author = {Cris Cunha and Wei Liu and Tim French and Ajmal Mian},
  title = {{Q-Cogni: An Integrated Causal Reinforcement Learning Framework}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2024},
  doi = {10.1109/tai.2024.3453230},
}
Figure