Comparing NARS and Reinforcement Learning: An Analysis of ONA and Q-Learning Algorithms

Ali Beikmohammadi, Sindri Magnússon

Abstract

Traditional Reinforcement Learning (RL) has achieved significant success in sequential decision-making tasks but often suffers from high data requirements and sensitivity to environmental stochasticity. This paper evaluates whether reasoning-based systems, specifically the Non-Axiomatic Reasoning System (NARS), can serve as a viable alternative. By comparing OpenNARS for Applications (ONA) against standard Tabular Q-Learning across several OpenAI Gym benchmarks, we demonstrate that while RL excels in stable, deterministic environments, ONA provides superior robustness and reliability in non-deterministic scenarios with significantly less hyperparameter tuning.

Introduction

Reinforcement Learning (RL) has become the dominant paradigm for training agents in robotics and gaming. However, standard RL algorithms often require millions of trials to converge on an optimal policy. This "data hunger" stems from a lack of compositional representation—the ability to logically combine learned sub-components of a problem to solve more complex tasks. Furthermore, RL agents frequently struggle to adapt when environmental conditions shift or when data is sparse.

This study explores an alternative approach rooted in Artificial General Intelligence (AGI) frameworks: the Non-Axiomatic Reasoning System (NARS). NARS is designed to operate under the Assumption of Insufficient Knowledge and Resources (AIKR), making it a candidate for more efficient and flexible learning. We benchmark ONA, a high-performance implementation of NARS, against a Tabular Q-Learning baseline to determine if a system built on formal logic can compete with one built on reward-maximization.

Algorithmic Frameworks

Tabular Q-Learning

As a model-free RL baseline, Tabular Q-Learning learns a state-action value function (Q-table). The agent updates these values based on observed rewards using the Bellman equation. To balance exploration and exploitation, the study utilizes an $\epsilon$-greedy strategy with an exponentially decaying exploration rate, ensuring the agent eventually converges toward an optimal policy in stable environments.

OpenNARS for Applications (ONA)

ONA implements Non-Axiomatic Logic (NAL), which differs fundamentally from the scalar reward system of RL. In ONA, knowledge is represented as statements with associated truth values, consisting of frequency (the proportion of positive evidence) and confidence (the amount of evidence supporting the claim).

Key features of the ONA implementation include:

  • Event-Based Perception: Observations are treated as discrete events in Narsese, a formal language.
  • Reasoning-Driven Action: The system uses forward and backward chaining to derive goals and beliefs.
  • Fallback Mechanisms: If logical reasoning does not suggest a specific action, the agent defaults to a random action to maintain task momentum without forcing a "no-op" state.

Experimental Methodology

The algorithms were evaluated using seven environments from the OpenAI Gym suite, categorized by their complexity and determinism:

  1. Navigation Tasks: CliffWalking-v0 and Taxi-v3.
  2. Stochastic Environments: FrozenLake-v1 (4x4 and 8x8), including "Slippery" variants where the agent has only a 1/3 chance of moving in the intended direction.
  3. Continuous-to-Discrete Mapping: FlappyBird-v0, requiring the translation of continuous coordinates into discrete Narsese statements.

Results and Discussion

Performance Trade-offs

The results highlight a clear divide in algorithmic strengths. Tabular Q-Learning outperformed ONA in CliffWalking-v0, Taxi-v3, and FlappyBird-v0. These environments are characterized by stable state-action mappings and consistent reward signals, which favor the iterative optimization process of RL.

However, ONA demonstrated a significant advantage in all FrozenLake-v1 variants, particularly the 8x8 Slippery version. In these non-deterministic environments, the logic-based updates of ONA allowed it to handle uncertainty more effectively than the Q-table, which can be easily disrupted by the noise of stochastic transitions.

Reliability and Stability

A critical finding of this study is the quantitative stability of the reasoning-based approach. Cumulative success plots revealed that ONA’s performance variance (standard deviation) was notably narrower than that of Q-Learning. This suggests that ONA is less dependent on "lucky" initializations or specific random seeds, making it a more reliable choice for consistent performance.

Hyperparameter Sensitivity

While Q-Learning is notoriously sensitive to hyperparameters such as learning rate, discount factors, and epsilon decay, ONA proved to be far more robust. By relying on the inherent properties of Non-Axiomatic Logic—where evidence naturally updates beliefs—the system required significantly less manual tuning to function across diverse tasks. Furthermore, ONA remained competitive even when provided only with sparse feedback (a single "goal achieved" signal at the end of an episode), whereas RL typically requires more frequent reward shaping.

Conclusion and Future Work

This analysis confirms that reasoning-based systems like ONA offer a compelling alternative to traditional RL, particularly in uncertain or data-scarce environments. While RL remains superior for stable, high-frequency reward tasks, ONA’s robustness and low hyperparameter sensitivity mark it as a powerful tool for AGI research.

Future work will focus on two primary areas: scaling and automation. Currently, continuous observations (as seen in FlappyBird) must be manually mapped to discrete Narsese statements. Developing automated methods for NARS to handle high-dimensional visual inputs is essential for scaling to more complex environments. Additionally, exploring hybrid models—where logical reasoning and statistical RL work in tandem—could combine the best of both worlds for next-generation intelligent agents.

Summary updated

BibTeX
@inproceedings{beikmohammadi2023comparing,
  author = {Ali Beikmohammadi and Sindri Magnússon},
  title = {{Comparing NARS and Reinforcement Learning: An Analysis of ONA and Q-Learning Algorithms}},
  booktitle = {AGI},
  year = {2023},
  doi = {10.1007/978-3-031-33469-6\_3},
}
Figure