Abstract
Current deep learning paradigms often treat model architectures and optimization algorithms as distinct entities, leading to static models that struggle with catastrophic forgetting and limited computational depth. This paper introduces Nested Learning (NL), a framework that unifies these components into a hierarchy of interconnected optimization problems. By reinterpreting layers and optimizers as associative memory modules, we develop the Hope architecture. Hope incorporates a Continuum Memory System (CMS) inspired by multi-timescale biological brain activity. Our results show that Hope maintains high performance across contexts of up to 10 million tokens and excels in continual learning tasks, providing a more transparent and adaptive approach to neural modeling.
Introduction
Despite the transformative impact of Large Language Models (LLMs), they remain hindered by three core limitations: a static nature that prevents seamless knowledge integration post-deployment, a "depth bottleneck" where additional layers do not necessarily equate to increased algorithmic complexity, and a conceptual divide between architecture and optimizer.
The Nested Learning (NL) framework posits that this heterogeneity is an "illusion." Drawing inspiration from the human brain—which utilizes uniform structures and multi-timescale updates—NL suggests that learning and architecture are fundamentally the same process: the management of associative memory.
The Nested Learning Framework
The NL paradigm represents machine learning models as sets of nested optimization problems, each maintaining its own "context flow." This approach provides a "white-box" view of model internals, contrasting with the often opaque gradient flows in traditional deep learning.
A key theoretical contribution of NL is the proof that standard components, such as Adam optimizers or Attention layers, are functionally equivalent to associative memory modules. For instance, the Adam optimizer can be viewed as a module designed to predict gradient variance. By unifying these elements, NL allows for the design of self-referential systems that can learn their own update rules in-context.
The Hope Architecture and CMS
To implement the NL framework, the authors introduced the Hope architecture. At its core is the Continuum Memory System (CMS), which replaces standard MLP blocks with a chain of modules updating at varying frequencies.
As shown in Figure 2, CMS is inspired by brain oscillations (delta, theta, alpha, beta, gamma). In this system, high-frequency neurons handle rapid adaptation to immediate context, while low-frequency neurons store persistent, long-term knowledge. Delta Gradient Descent (DGD) is employed as a more expressive update rule, utilizing $L_2$ regression to improve memory retrieval over simple dot-product similarity. This design creates a robust memory loop; information lost by fast-updating blocks can be recovered from slower blocks via backpropagation, effectively resisting catastrophic forgetting.
Experimental Evaluation
The Hope architecture was evaluated across a diverse set of benchmarks focusing on long-context reasoning and continual learning.
Long-Context Performance
On the BABILong benchmark, Hope demonstrated the ability to process up to 10 million tokens while maintaining high accuracy, significantly outperforming established models like Llama-3 and other recurrent architectures. This suggests that the multi-timescale approach of CMS effectively manages extremely long dependencies.
Continual Learning and Reasoning
In class-incremental tasks (using datasets like CLINC and Banking), Hope achieved superior accuracy compared to traditional methods like Elastic Weight Consolidation (EWC). Furthermore, in zero-shot reasoning tasks across 10 common-sense benchmarks (e.g., PIQA, HellaSwag), Hope consistently outperformed standard Transformers and hybrid models at scales up to 1.3B parameters.
Conclusion and Future Work
Nested Learning offers a path toward more adaptive and interpretable AI by exploiting the "extra design axis" of nested optimization levels. While the Hope architecture shows significant promise, challenges remain in scaling these expressive optimizers to the 70B+ parameter range. By moving away from static, layered stacks toward dynamic, nested hierarchies, we can better approximate the flexible learning capabilities of biological systems.
Summary updated