Introduction
Reinforcement learning (RL) has traditionally been divided into model-free approaches, which learn habits through trial and error, and model-based approaches, which plan using environmental maps. While model-free agents are computationally efficient, they often suffer from extreme sample inefficiency and lack interpretability, operating as "black boxes." A fundamental limitation in current RL is the lack of causal understanding; agents may correlate actions with rewards without grasping the underlying "why."
Q-Cogni addresses this gap by bridging the divide between habit-based learning and causal reasoning. It is a hybrid RL framework that autonomously discovers the causal structure of an environment to guide the learning process. By integrating a Bayesian Network into a redesigned Q-Learning algorithm, Q-Cogni achieves superior sample efficiency, scalability for high-dimensional routing, and human-readable decision-making.
Methodology
The Q-Cogni framework is composed of three primary modules designed to mimic the human brain's dual-system approach to learning and reasoning.
1. Autonomous Causal Structure Discovery
The process begins with an agent performing a random walk to gather initial environmental data. Using the NOTEARS algorithm, the module identifies a Directed Acyclic Graph (DAG) that represents the causal relationships between states, actions, and rewards. This step transforms raw observational data into a structured causal model.
2. Causal Inference Module
The discovered DAG is converted into a Bayesian Network. This allows the agent to perform causal queries, calculating the probability of achieving specific sub-goals based on current states and potential actions. For example, the agent can determine the likelihood that a specific movement will lead to a successful passenger pickup.
3. Modified Q-Learning
The core of the framework is a redesigned Q-Learning algorithm. Instead of relying solely on standard exploration strategies like $\epsilon$-greedy, Q-Cogni uses the Bayesian Network to "shortcut" exploration. If the causal model identifies an action as a direct "parent" of a goal, the agent prioritizes it. Additionally, the Q-function incorporates probabilistic reward adjustments to facilitate learning in environments with sparse rewards.
Experimental Setup
The framework was validated across three distinct environments:
- Taxi-v3 (OpenAI Gym): A benchmark $5 \times 5$ grid task involving passenger pickup and delivery.
- Synthetic Large-Scale Grids: Scalability tests ranging from $8 \times 8$ to $512 \times 512$ grids.
- Real-World NYC TLC Data: Using the OSMNX library, the authors modeled the Astoria region of New York City as a graph with 666 nodes and 1,712 edges to test navigation in complex, real-world topologies.
Results and Discussion
Experimental results demonstrate that Q-Cogni significantly outperforms standard algorithms like PPO and DDQN in terms of both efficiency and stability. In the Taxi-v3 environment, the agent reached optimal rewards faster and with lower variance.
A key finding is the framework's scalability. While traditional shortest-path algorithms like Dijkstra and A* become computationally expensive as node counts increase, Q-Cogni scales near-linearly. In the NYC dataset, Q-Cogni's policies matched or exceeded traditional search methods in 85% of cases.
Furthermore, the framework offers high causal interpretability. Unlike black-box models, Q-Cogni can provide conditional probabilities for its actions, explaining why a specific move was chosen based on its predicted causal impact on the goal. Finally, the causal structure discovered in small environments proved to be domain-agnostic, allowing it to be transferred to larger, real-world maps without retraining the discovery module.
Conclusion and Future Work
Q-Cogni represents a significant step toward integrating causal reasoning into reinforcement learning. By leveraging discovered causal structures, it overcomes traditional limitations in sample efficiency and interpretability. While currently optimized for discrete state-action spaces, future research will focus on extending this framework to continuous variables, which are essential for broader applications in robotics and autonomous control.
Summary updated