Linking Rapid Learning to Compositional Structure in Neural Networks: Evidence from Additive Analogies in Meta-Learned Parameter Spaces

Abigail L. Tenenbaum, R. Thomas McCoyarXiv 2026 · arXiv

In the quest to bridge the gap between human-like "one-shot" learning and the data-hungry nature of deep learning, a new paper from researchers at Yale University offers a fascinating geometric explanation. By treating the entire set of a neural network's weights as a single point in a high-dimensional space, they show that meta-learning organizes this space into a "word2vec for functions."

TL;DR

This paper demonstrates that meta-learning (MAML) organizes a neural network's parameter space such that different functions can be added and subtracted like vectors (e.g., Function A - Feature 1 + Feature 2 = Function B). This emergent geometric structure allows networks to implement entirely new functions through simple arithmetic on their weights, providing a concrete "geometric basis" for how inductive biases enable rapid learning.

What Problem Does This Paper Address?

Humans are remarkably efficient learners; we can often grasp a new concept or linguistic rule from just one or two examples. In contrast, standard neural networks typically require thousands or millions of training steps to achieve the same.

Cognitive science has long debated how to reconcile symbolic models (which are structured and learn fast) with connectionist models (neural networks, which are continuous and learn slowly). While meta-learning—"learning to learn"—has successfully produced networks that adapt rapidly, the internal mechanism remains a "black box." The authors ask: What is the internal structure of a meta-learned network that actually enables this speed?

Methods

The researchers focused on Model-Agnostic Meta-Learning (MAML). MAML doesn't just train a network to perform a task; it trains the network's initialization so that it can reach a solution for any task in a given domain in just one or two gradient steps.

To probe the internal structure, they used Additive Analogies. Taking inspiration from the famous word embedding result where $king - man + woman \approx queen$, they applied this to "function embeddings." 1. They meta-learned an initialization $\theta$. 2. They adapted $\theta$ to several specific functions (e.g., $f, g, p$), resulting in adapted parameter vectors $\theta_f, \theta_g, \theta_p$. 3. They performed vector arithmetic: $W = \theta_p + \theta_g - \theta_f$. 4. They tested if the network defined by the resulting weights $W$ actually implemented the predicted fourth function $q$, without any training on $q$.

The full analogy pipeline: from meta-learning an initialization to constructing new functions via vector arithmetic.

Datasets

The authors used three synthetic, highly structured sequence-generation domains to isolate different types of learning:

  • ABgeom: Purely lexical variation. Sequences like "5 42 5 42..." where the tokens $A$ and $B$ change, but the alternating structure is constant.
  • ABparity: Syntactic variation. Rules based on whether the count of tokens is even or odd.
  • ABlength: Syntactic variation. Rules based on the specific number of "filler" tokens between $A$ and $B$.

They tested these across two major architectures: Transformers and LSTMs.

Key Insights and Ideas

  • Inductive Bias as Geometry: The paper argues that inductive bias isn't just a "good starting point," but a systematic organization of the trajectories available from that point. Rapid learning happens because the "map" of parameter space is laid out logically.
  • Emergent Compositionality: Even though the networks were never explicitly told to be "compositional," the pressure to learn rapidly forced them to organize their weights such that symbolic components (like "parity" or "token value") became independent, additive directions in parameter space.
  • Multiple Realizability: Interestingly, different architectures might "choose" different geometric representations for the same symbolic rule. For example, in the ABlength domain, Transformers and LSTMs often converged on different syntactic representations (e.g., "first-total" vs. "first-last" counts), yet both were internally consistent and supported analogies.

Key Results

The results were strikingly robust. For meta-learned networks, the "analogy" networks (created by weight arithmetic) performed almost perfectly, matching the performance of networks that were actually trained on the data.

  • MAML vs. The Rest: Meta-learned initializations achieved mean F-scores of ~1.0 on analogies. In contrast, randomly initialized networks and even conventionally pre-trained networks failed to show this robust geometric structure (F-scores < 0.1 in many rapid-adaptation cases).
  • Robustness: This structure emerged across both Transformers and LSTMs, suggesting it is a fundamental property of the meta-learning objective rather than a specific architectural quirk.
Performance across architectures and domains. "Analogy: mean-max" shows that when we test the right syntactic hypothesis, meta-learned networks exhibit near-perfect geometric structure.

As shown in Figure 5, there is a clear hierarchy: MAML produces the most robust geometry, followed by pre-training, with random initialization providing almost no useful geometric structure for rapid adaptation.

Comparison of initialization conditions. MAML-tuned networks significantly outperform pre-trained and random baselines in supporting additive analogies.

Limitations and Future Work

  • Synthetic Domains: The study uses simple, controlled sequence tasks. Whether this "parameter-space geometry" holds for massive, real-world tasks like full-scale natural language processing remains to be seen.
  • Hypothesis Testing: Currently, the researcher must guess the "symbolic representation" to write the analogy. Future work could involve developing methods to automatically discover the geometric "basis" of a parameter space.
  • Causality: While the correlation between rapid learning and geometric structure is clear, the authors note that proving the geometry causes the rapid learning (rather than being a byproduct) is an important next step.

By showing that "symbolic" relationships can emerge as "geometric" ones, this work provides a promising path toward neural networks that are not just powerful, but also as efficient and systematic as the human mind.

Summary updated

Abstract

Humans can learn rapidly from limited experience, while neural networks typically require extensive training. In this work, we investigate what properties of a neural network can enable it to learn rapidly. We study neural networks that have been optimized via meta-learning, a technique that exposes a network to many related functions to facilitate its subsequent learning of similar functions. Prior work has shown that these meta-learned networks can learn new functions rapidly, but how they do so remains unclear. To investigate their internal structure, we take inspiration from classic work demonstrating additive relationships among the learned vectors that neural networks use to represent words (e.g., "king" minus "man" plus "woman" approximates "queen"). Analogously, we look for additive relationships among a meta-learned network's "representations" of different functions; when a meta-learned network rapidly learns to implement a specific function, all of its adapted parameter values can be concatenated into a single vector that we treat as representing the function. We find that such parameter vectors can be added and subtracted according to symbolic relationships among the functions they represent, yielding new parameter vectors that successfully represent the functions predicted by the corresponding symbolic manipulations: the networks specified by these artificially recombined vectors actually implement the intended functions without any direct training on them. Our results suggest that meta-learning can tune a network's starting point to organize subsequent learning along systematic, functionally meaningful trajectories through the space of possible parameter vectors, providing a geometric basis for realizing inductive biases in neural networks.
BibTeX
@misc{tenenbaum2026linking,
  author = {Abigail L. Tenenbaum and R. Thomas McCoy},
  title = {{Linking Rapid Learning to Compositional Structure in Neural Networks: Evidence from Additive Analogies in Meta-Learned Parameter Spaces}},
  year = {2026},
  eprint = {2611.17314},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17314},
}
Figure