Family Discovery

Stephen M. OmohundroNeurIPS 1995

This paper, published in the mid-90s by Stephen M. Omohundro, introduces a framework for learning the underlying structure of related models, a process he calls "Family Discovery." It serves as a foundational look at what we might today call meta-learning or manifold learning for model parameters.

TL;DR

"Family Discovery" is a learning paradigm that identifies the low-dimensional structure (the "family") connecting different but related tasks, such as different speakers' accents or different document fonts. By learning this structure, a system can generalize better to new tasks even with very little data, significantly outperforming standard "one-size-fits-all" or "learn-from-scratch" approaches.

What Problem Does This Paper Address?

The author identifies a gap in how machine learning systems handle variation. Consider speech recognition:

  1. The "Single Model" approach: You train one massive model on everyone’s speech. This often results in a "blurry" model that confuses words because it ignores individual accents.
  2. The "Separate Models" approach: You train a unique model for every single speaker. This fails because it doesn't allow the system to use what it learned from Speaker A to help understand Speaker B.

The challenge is to find the "Goldilocks" zone: learning that while every speaker is different, they all exist within a structured "space of accents." If we can map this space (the "family"), we can identify a new speaker's position in that space and instantly have a highly accurate, personalized model.

Methods

The paper proposes an iterative, EM-like (Expectation-Maximization) algorithm. First, it fits models to individual "episodes" (e.g., speech from one person). Then, it finds a low-dimensional surface (the family) that passes near these model parameters. Finally, it re-adjusts the individual models to move closer to this surface, effectively using the "family" as a prior.

Three family representations: Affine, Affine Patch, and Coupled Map

Omohundro compares three specific family representations to capture this structure:

  • Affine Algorithm: The simplest version, where the family is a linear subspace (found via Principal Component Analysis).
  • Affine Patch Algorithm: A "surface learning" approach where the family is represented by a collection of local linear patches blended together with Gaussians.
  • Coupled Map Algorithm: A more complex nonlinear approach using a pair of maps—one to "encode" model parameters into a low-dimensional space and another to "decode" them back. This is implemented using a "mixture of experts" architecture.

Datasets

The paper uses a synthetic classification task to benchmark the algorithms:

  • Task: Two-class classification in a 5-dimensional feature space.
  • The "Family": The means of the classes are determined by a nonlinear, two-parameter family (representing the "underlying traits" of the episode).
  • Structure: The training data is partitioned into "episodes" (varying from 200 to 2000 total examples, split into $\sqrt{total}$ episodes).
  • Evaluation: Performance is measured by the error rate on an independent test set of 50 new episodes.

Key Insights and Ideas

  • Episodes as Information Units: The core insight is that data isn't just a flat list; it’s often grouped. Knowing that a set of samples came from the same underlying source (the same speaker or the same font) is a powerful signal that standard models often throw away.
  • The Family Prior: By treating the learned manifold as a "prior," the system can perform "transfer learning." An episode with very little data can "borrow" strength from other episodes because the system knows it must lie on the same family surface.
  • Nonlinearity Matters: The paper demonstrates that while linear (Affine) models are easier to implement, they fail to capture the true structure of complex tasks. The "Affine Patch" and "Coupled Map" approaches are necessary when the relationship between models is curved or topologically complex.

Key Results

The empirical results show a clear hierarchy of performance across the different modeling strategies:

Performance comparison of different modeling approaches
  • Worst: The Single Model approach had the highest error rates (approaching 50% in some cases), as it couldn't handle the shifting classification boundaries.
  • Better: The Separate Models approach improved as more data became available but remained inefficient.
  • Best: The Family Discovery algorithms (Patch and Map) consistently achieved the lowest error rates (dropping to ~33-35%).
  • Linear vs. Nonlinear: Because the test task was nonlinear, the nonlinear "Patch" and "Map" algorithms significantly outperformed the linear "Affine" algorithm. Interestingly, there was no clear winner between the Patch and Map approaches, suggesting both are viable for capturing complex manifolds.

Limitations and Future Work

  • Pre-partitioned Data: The primary experiments assume the "teacher" has already labeled which data belongs to which episode. While the paper suggests using clustering or EM to discover these boundaries automatically, this adds significant complexity.
  • Computational Cost: The iterative nature of fitting individual models and then fitting the family surface is computationally more expensive than standard training.
  • Parameter Space Dimension: The paper tests on a 5D feature space. Scaling "Family Discovery" to the massive parameter spaces of modern deep learning (millions of weights) remains a significant challenge that the paper does not address.

Summary updated

BibTeX
@inproceedings{omohundro1995family,
  author = {Stephen M. Omohundro},
  title = {{Family Discovery}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1995},
}
Figure