Nested Learning: The Illusion of Deep Learning Architectures

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni

Abstract

Current deep learning paradigms often treat model architectures and optimization algorithms as distinct entities, leading to static models that struggle with catastrophic forgetting and limited computational depth. This paper introduces Nested Learning (NL), a framework that unifies these components into a hierarchy of interconnected optimization problems. By reinterpreting layers and optimizers as associative memory modules, we develop the Hope architecture. Hope incorporates a Continuum Memory System (CMS) inspired by multi-timescale biological brain activity. Our results show that Hope maintains high performance across contexts of up to 10 million tokens and excels in continual learning tasks, providing a more transparent and adaptive approach to neural modeling.

Introduction

Despite the transformative impact of Large Language Models (LLMs), they remain hindered by three core limitations: a static nature that prevents seamless knowledge integration post-deployment, a "depth bottleneck" where additional layers do not necessarily equate to increased algorithmic complexity, and a conceptual divide between architecture and optimizer.

The Nested Learning (NL) framework posits that this heterogeneity is an "illusion." Drawing inspiration from the human brain—which utilizes uniform structures and multi-timescale updates—NL suggests that learning and architecture are fundamentally the same process: the management of associative memory.

The Nested Learning Framework

The NL paradigm represents machine learning models as sets of nested optimization problems, each maintaining its own "context flow." This approach provides a "white-box" view of model internals, contrasting with the often opaque gradient flows in traditional deep learning.

Comparison between traditional Deep Learning and the transparent Nested Learning Module.

A key theoretical contribution of NL is the proof that standard components, such as Adam optimizers or Attention layers, are functionally equivalent to associative memory modules. For instance, the Adam optimizer can be viewed as a module designed to predict gradient variance. By unifying these elements, NL allows for the design of self-referential systems that can learn their own update rules in-context.

The Hope Architecture and CMS

To implement the NL framework, the authors introduced the Hope architecture. At its core is the Continuum Memory System (CMS), which replaces standard MLP blocks with a chain of modules updating at varying frequencies.

Biological inspiration for the Continuum Memory System (CMS) and its multi-timescale update structure.

As shown in Figure 2, CMS is inspired by brain oscillations (delta, theta, alpha, beta, gamma). In this system, high-frequency neurons handle rapid adaptation to immediate context, while low-frequency neurons store persistent, long-term knowledge. Delta Gradient Descent (DGD) is employed as a more expressive update rule, utilizing $L_2$ regression to improve memory retrieval over simple dot-product similarity. This design creates a robust memory loop; information lost by fast-updating blocks can be recovered from slower blocks via backpropagation, effectively resisting catastrophic forgetting.

Experimental Evaluation

The Hope architecture was evaluated across a diverse set of benchmarks focusing on long-context reasoning and continual learning.

Long-Context Performance

On the BABILong benchmark, Hope demonstrated the ability to process up to 10 million tokens while maintaining high accuracy, significantly outperforming established models like Llama-3 and other recurrent architectures. This suggests that the multi-timescale approach of CMS effectively manages extremely long dependencies.

Continual Learning and Reasoning

In class-incremental tasks (using datasets like CLINC and Banking), Hope achieved superior accuracy compared to traditional methods like Elastic Weight Consolidation (EWC). Furthermore, in zero-shot reasoning tasks across 10 common-sense benchmarks (e.g., PIQA, HellaSwag), Hope consistently outperformed standard Transformers and hybrid models at scales up to 1.3B parameters.

Conclusion and Future Work

Nested Learning offers a path toward more adaptive and interpretable AI by exploiting the "extra design axis" of nested optimization levels. While the Hope architecture shows significant promise, challenges remain in scaling these expressive optimizers to the 70B+ parameter range. By moving away from static, layered stacks toward dynamic, nested hierarchies, we can better approximate the flexible learning capabilities of biological systems.

Summary updated

Abstract

Over the last decades, developing more powerful neural architectures and simultaneously designing optimization algorithms to effectively train them have been the core of research efforts to enhance the capability of machine learning models. Despite the recent progresses, particularly in developing Language Models (LMs), there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improved, and find ''effective solutions,''. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own ''context flow''. NL reveals that existing deep learning methods learns from data through \emph{compressing} their own context flow, and explain how in-context learning emerges in large models. NL suggests a path (a new dimension to deep learning) to design more expressive learning algorithms with more ''levels'', resulting in higher-order in-context learning abilities. In addition to its neuroscientifically plausible and mathematically white-box nature, we advocate for its importance by presenting three core contributions: (1) Deep Optimizers: Based on NL, we show that well-known gradient-based optimizers (e.g., Adam, SGD with Momentum, etc.) are in fact associative memory modules that aim to compress the gradients with gradient descent. Building on this insight, we present a set of more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Titans: Taking advantage of NL's insights on learning algorithms, we present a novel sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of ``long-term/short-term memory''. Combining our self-modifying sequence model with the continuum memory system, we present a learning module, called Hope, showing promising results in language modeling, continual learning, and long-context reasoning tasks.
BibTeX
@inproceedings{behrouz2025nested,
  author = {Ali Behrouz and Meisam Razaviyayn and Peilin Zhong and Vahab Mirrokni},
  title = {{Nested Learning: The Illusion of Deep Learning Architectures}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=nbMeRvNb7A},
}
Figure