Distilling a Modular Reservoir Through a Genomic Bottleneck

Mani Hamidi, Sina Khajehabdollahi, Charley M. Wu, Emmanouil GiannakakisarXiv 2026 · arXiv

TL;DR

Inspired by how biological evolution compresses structural blueprints into a compact genome, this paper introduces an "evo-devo" approach to reservoir computing. By using a tiny hypernetwork to generate the inter-module connections of a growing, modular recurrent neural network, the authors achieve massive parameter savings, flat scaling, and significantly enhanced robustness against noise without sacrificing performance.

What Problem Does This Paper Address?

Training large recurrent neural networks (RNNs) from scratch is notoriously parameter-heavy and computationally expensive. As tasks grow more complex or network modules scale up, the number of trainable parameters typically explodes quadratically.

In contrast, biological organisms don't start from a blank slate. Nature compresses millions of years of structural learning into a compact "genomic bottleneck"—the genome. This genetic blueprint guides development to pre-wire functional, modular brain structures before any lifelong learning or experience-dependent plasticity even begins. The authors address the challenge of bringing this extreme structural compression and efficiency to artificial neural networks, specifically within the framework of reservoir computing.

Methods

The authors propose a two-level learning scheme that mimics evolution and development:

  1. The Phenotypic Network (p-net): A hierarchical modular reservoir computer that grows over time to handle increasingly complex tasks via a curriculum. Each module consists of fixed internal input ($W^{\text{IN}}$) and recurrent ($W^{\text{R}}$) weights that are duplicated as new modules are added.
  2. The Genomic Network (g-net): A highly compressed hypernetwork (a simple 3-layer Multi-Layer Perceptron with just 16 neurons per layer) that acts as the "genome." Instead of training the feedforward connections ($W^{\text{F}}$) between the p-net's modules directly, the g-net takes the source neuron index, target neuron index, and module index as inputs and generates the weights.

During training, backpropagation updates the parameters of the tiny g-net and the linear readouts of the p-net, rather than modifying the massive p-net weight matrices directly. This is compared against a traditional "direct training" baseline where the inter-module weights are learned traditionally via backpropagation.

Implementing an indirect learning scheme using a g-net

Datasets

The models are evaluated on two classic working-memory benchmarks that scale systematically in difficulty:

  • N-parity: The network receives a binary stream and must output the parity (binary sum) of the last $n$ bits for all lengths up to $N$.
  • N-Delayed Match to Sample (N-DMS): The network must determine if the current input bit matches the bit presented exactly $n$ steps ago, for all $1 \le n \le N$.

No new datasets were introduced; these are standard, rigorous benchmarks for testing working-memory capacity and temporal processing in RNNs.

Key Insights and Ideas

  • The Genomic Bottleneck as a Regularizer: Forcing a tiny hypernetwork to generate large weight matrices acts as a powerful, task-agnostic regularizer. It prevents overfitting by compelling the network to discover smooth, continuous, and highly structured connectivity patterns.
  • Decoupling Size from Complexity: In traditional architectures, larger modules mean a quadratic explosion of trainable parameters ($\mathcal{O}(NM^2)$). With the g-net approach, the number of trainable parameters remains virtually constant regardless of how large the modules are or how deep the network grows.
  • Emergent Low-Rank Structure: Singular Value Decomposition (SVD) revealed that the generated weights are inherently low-rank and highly compressible, both across modules (reusing patterns) and within individual modules. This property directly mirrors biological structural efficiency.

Key Results

  • Parameter Scaling: For direct training, parameters scale quadratically with module size ($M$) and linearly with task difficulty ($N$). For the indirect g-net approach, the parameter count remains flat and low across all conditions.
  • Compressibility: SVD analysis showed that g-net generated weights require only 2 to 6 components to achieve 90% reconstruction fidelity across 30 modules, compared to over 25 components for directly trained weights.
  • Robustness: When weights were perturbed with random noise, the g-net networks showed significantly higher accuracy retention, particularly at smaller module sizes ($M < 60$), demonstrating a superior efficiency-robustness trade-off.
Network robustness and parameter efficiency under perturbation

Limitations and Future Work

  • Sensitivity to Initialization: Hypernetworks can be highly sensitive to initial conditions. The authors noted that some random initializations caused an early lag or delay before the g-net successfully entered the curriculum-advancing regime, whereas direct training is more stable initially.
  • Moving Beyond Backpropagation: Currently, the authors use backpropagation to train the g-net. True biological evolution relies on non-gradient methods. The authors note that a major focus of future direction will be to shift towards neuroevolutionary learning procedures (like genetic algorithms or sexual recombination) to optimize the g-net more faithfully.

Summary updated

Abstract

The intricate structures of biological neural networks largely emerge during development, guided by a comparatively compressed blueprint encoded in the genome. The connectivity that emerges from this decoding process is rich in structure, and already equips the organism with functional modules upon birth. This initial structure serves as a scaffold that can be gradually refined and fine-tuned through lifelong experience, via a variety of plasticity mechanisms. Drawing inspiration from this interaction between evolutionary and developmental modes of learning, we use hypernetworks to learn a compressed generative process that generates the connectivity of a modular reservoir. We show that this marriage between curriculum-based meta-learning and modular reservoir computing can generate sparse recurrent networks that solve difficult temporal tasks with minimal training and without concessions to robustness.
BibTeX
@misc{hamidi2026distilling,
  author = {Mani Hamidi and Sina Khajehabdollahi and Charley M. Wu and Emmanouil Giannakakis},
  title = {{Distilling a Modular Reservoir Through a Genomic Bottleneck}},
  year = {2026},
  eprint = {2606.28380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.28380},
}
Figure