Abstract
TD-FALCON is a self-organizing neural network that incorporates Temporal Difference (TD) methods for reinforcement learning. Despite the advantages of fast and stable learning, TD-FALCON still relies on an iterative process to evaluate each available action in a decision cycle. To remove this deficiency, this paper presents a direct code access procedure whereby TD-FALCON conducts instantaneous searches for cognitive nodes that match with the current states and at the same time provide maximal reward values. Our comparative experiments show that TD-FALCON with direct code access produces comparable performance with the original TD-FALCON while improving significantly in computation efficiency and network complexity.
Introduction: Overcoming the Iterative Bottleneck
Reinforcement Learning (RL) agents often struggle with the "curse of dimensionality." In environments with massive or continuous state and action spaces, traditional methods become computationally expensive. The authors previously developed TD-FALCON, a neural architecture that learns stable, online mappings between states, actions, and rewards.
However, the original TD-FALCON had a "brute force" bottleneck: to choose the best move, it had to enumerate and evaluate every possible action one by one. This paper introduces a "direct code access" mechanism to move from "evaluating every option" to "instantly finding the best option," achieving a 40x speedup and significantly reduced network complexity without sacrificing performance.
The FALCON Architecture and Direct Code Access
The core innovation is the Direct Code Access procedure, built upon the FALCON (Fusion Architecture for Learning, COgnition, and Navigation) framework. FALCON is a 3-channel self-organizing map based on Adaptive Resonance Theory (ART).
Instead of cycling through actions, the system initializes a search by setting the State channel to the current environment, the Action channel to a "don't care" state (all 1s), and the Reward channel to the maximum possible value (1.0). Using the mathematical properties of Fuzzy ART, the network performs a bottom-up propagation to find the winning "cognitive node" that most closely matches the current state while representing the highest predicted reward. This treats action selection as a pattern completion problem rather than a repetitive calculation.
Experimental Evaluation: Minefield Navigation
The approach was evaluated using a Minefield Navigation Task in a 16x16 grid containing stationary mines and a target. An autonomous agent equipped with sonar and target-bearing sensors must navigate to the target without hitting mines or exceeding 30 steps. The new Direct Access (DA) model was compared against the original Action Enumeration (AE) model and a standard Backpropagation (BP) reinforcement learner.
Key Results and Insights
The experiments demonstrated that the Direct Access method provides a massive speedup, with decision cycle times dropping from approximately 4.5ms to just 0.12ms. Furthermore, the DA network is highly memory-efficient, functioning effectively with a significantly smaller set of cognitive nodes (around 38) compared to the iterative version.
As shown in the performance curves, both FALCON models reached over 90% success rates within 1,000 trials. In stark contrast, the BP-based learner struggled to exceed 10% in the same timeframe, requiring significantly more exploration to reach even basic competency. This highlights the stability and speed of ART-based self-organization over traditional gradient-descent methods.
Conclusion and Future Work
By replacing slow, iterative action evaluation with instantaneous pattern matching, Direct Code Access makes TD-FALCON viable for more complex, real-time tasks. While the current validation used discrete actions, the architecture is theoretically applicable to continuous action spaces. Future work will focus on testing in truly continuous control tasks, such as robotic manipulation, and exploring the interpretability of the learned cognitive rules.
Summary updated