Direct Code Access in Self-Organizing Neural Networks for Reinforcement Learning

Ah-Hwee TanIJCAI 2007 · OpenReview

Abstract

TD-FALCON is a self-organizing neural network that incorporates Temporal Difference (TD) methods for reinforcement learning. Despite the advantages of fast and stable learning, TD-FALCON still relies on an iterative process to evaluate each available action in a decision cycle. To remove this deficiency, this paper presents a direct code access procedure whereby TD-FALCON conducts instantaneous searches for cognitive nodes that match with the current states and at the same time provide maximal reward values. Our comparative experiments show that TD-FALCON with direct code access produces comparable performance with the original TD-FALCON while improving significantly in computation efficiency and network complexity.

Introduction: Overcoming the Iterative Bottleneck

Reinforcement Learning (RL) agents often struggle with the "curse of dimensionality." In environments with massive or continuous state and action spaces, traditional methods become computationally expensive. The authors previously developed TD-FALCON, a neural architecture that learns stable, online mappings between states, actions, and rewards.

However, the original TD-FALCON had a "brute force" bottleneck: to choose the best move, it had to enumerate and evaluate every possible action one by one. This paper introduces a "direct code access" mechanism to move from "evaluating every option" to "instantly finding the best option," achieving a 40x speedup and significantly reduced network complexity without sacrificing performance.

The FALCON Architecture and Direct Code Access

The core innovation is the Direct Code Access procedure, built upon the FALCON (Fusion Architecture for Learning, COgnition, and Navigation) framework. FALCON is a 3-channel self-organizing map based on Adaptive Resonance Theory (ART).

The FALCON architecture featuring Sensory (State), Motor (Action), and Feedback (Reward) fields.

Instead of cycling through actions, the system initializes a search by setting the State channel to the current environment, the Action channel to a "don't care" state (all 1s), and the Reward channel to the maximum possible value (1.0). Using the mathematical properties of Fuzzy ART, the network performs a bottom-up propagation to find the winning "cognitive node" that most closely matches the current state while representing the highest predicted reward. This treats action selection as a pattern completion problem rather than a repetitive calculation.

Experimental Evaluation: Minefield Navigation

The approach was evaluated using a Minefield Navigation Task in a 16x16 grid containing stationary mines and a target. An autonomous agent equipped with sonar and target-bearing sensors must navigate to the target without hitting mines or exceeding 30 steps. The new Direct Access (DA) model was compared against the original Action Enumeration (AE) model and a standard Backpropagation (BP) reinforcement learner.

Key Results and Insights

The experiments demonstrated that the Direct Access method provides a massive speedup, with decision cycle times dropping from approximately 4.5ms to just 0.12ms. Furthermore, the DA network is highly memory-efficient, functioning effectively with a significantly smaller set of cognitive nodes (around 38) compared to the iterative version.

Success rates of FALCON-AE and FALCON-DA compared to a standard Backpropagation (BP) learner.

As shown in the performance curves, both FALCON models reached over 90% success rates within 1,000 trials. In stark contrast, the BP-based learner struggled to exceed 10% in the same timeframe, requiring significantly more exploration to reach even basic competency. This highlights the stability and speed of ART-based self-organization over traditional gradient-descent methods.

Conclusion and Future Work

By replacing slow, iterative action evaluation with instantaneous pattern matching, Direct Code Access makes TD-FALCON viable for more complex, real-time tasks. While the current validation used discrete actions, the architecture is theoretically applicable to continuous action spaces. Future work will focus on testing in truly continuous control tasks, such as robotic manipulation, and exploring the interpretability of the learned cognitive rules.

Summary updated

Abstract

TD-FALCON is a self-organizing neural network that incorporates Temporal Difference (TD) methods for reinforcement learning. Despite the advantages of fast and stable learning, TD-FALCON still relies on an iterative process to evaluate each available action in a decision cycle. To remove this deficiency, this paper presents a direct code access procedure whereby TD-FALCON conducts instantaneous searches for cognitive nodes that match with the current states and at the same time providemaximal reward values. Our comparative experiments show that TD-FALCON with direct code access produces comparable performance with the original TD-FALCON while improving significantly in computation efficiency and network complexity.
BibTeX
@inproceedings{tan2007direct,
  author = {Ah-Hwee Tan},
  title = {{Direct Code Access in Self-Organizing Neural Networks for Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
  url = {https://openreview.net/forum?id=rkZE54z_WS},
}
Figure