Imagine a child is shown three pictures of Dalmatians and told, "Look at these feps." What does the word fep mean? It could mean "Dalmatian," "dog," "canine," or "animal." Mathematically, all of these hypotheses are perfectly consistent with the observed data. Yet, humans instantly and reliably guess that fep refers to the most specific category—Dalmatian—rather than over-generalizing to all animals.
How do intelligent agents achieve this without negative examples (e.g., being told a cat is not a fep)? In cognitive science and machine learning, explaining how learners infer the correct level of abstraction from sparse, positive-only data has long been a fundamental challenge. As explored by Peterson et al., understanding this capacity requires a principle that naturally weights more specific hypotheses more heavily during induction.
The Mathematics of Coincidence
The mechanism that solves this puzzle is the size principle. The core intuition is simple: we treat the data we observe not just as a set of arbitrary facts, but as a random sample from the true concept.
As formulated by Tenenbaum in Bayesian Modeling of Human Concept Learning, if a hypothesis $H$ is true, and we observe $n$ independent examples sampled at random from its extension, the likelihood of observing those specific data points $D$ is inversely proportional to the size of the hypothesis's extension, raised to the power of $n$:
$$P(D|H) = \left(\frac{1}{|H|}\right)^n$$
Why does this mathematical formulation work so well? Consider our Dalmatian example. If the true meaning of fep were "animal" (a very large set), drawing three examples at random and happening to get three Dalmatians would be an extraordinary statistical coincidence. But if the true meaning were "Dalmatian" (a much smaller set), drawing three Dalmatians is highly probable.
As Sanjana et al. point out, the size principle acts as a "Bayesian Occam's Razor." It penalizes overly broad hypotheses because their probability mass is spread too thin. The smaller and more specific the hypothesis, the higher its likelihood—provided it still encompasses all the observed data.
Bridging Rules and Similarity
One of the most elegant consequences of the size principle is how it naturally resolves a classic dichotomy in cognitive psychology: the debate between similarity-based and rule-based learning. Historically, these were viewed as separate systems—one fuzzy and associative, the other rigid and logical.
In Rules and Similarity in Concept Learning, Tenenbaum demonstrates that both behaviors emerge from a single Bayesian framework. The shift depends entirely on the amount of data observed:
- Sparse Data (Fuzzy Similarity): When you have only seen a single example, the likelihood term $(1/|H|)^1$ has moderate influence. The learner's generalization is dominated by their prior distribution, which typically favors smooth, similarity-based gradients. The boundaries of the concept appear "fuzzy."
- Accumulating Data (Sharp Rules): As the number of examples $n$ increases, the likelihood term $\left(1/|H|\right)^n$ grows exponentially dominant. It swiftly crushes broader hypotheses, forcing the posterior distribution to concentrate entirely on the narrowest possible category that fits the data. The generalization boundary abruptly sharpens, mimicking a strict logical rule.
This transition from graded similarity to sharp categorization as data accumulates is a hallmark of human learning, seen clearly in phenomena like "fast mapping" in child language acquisition, as documented by Xu.
Generalization Across Domains
The size principle is not just a theoretical convenience for toy problems; it is a foundational property of how intelligent agents navigate the world. Meder et al. highlight how this principle enables rapid knowledge transfer and efficient hypothesis selection even when data is highly sparse, unifying diverse generalization tasks under a shared probabilistic umbrella.
Furthermore, the principle scales effectively to complex, high-dimensional machine learning tasks. For instance, Jia et al. integrated the size principle with probabilistic image classifiers to solve hierarchical visual concept learning. By evaluating the size of categories within a semantic hierarchy, their system can automatically determine whether a cluster of images represents a specific breed, a general species, or a broad biological kingdom—all without ever requiring negative labels.
Ultimately, as demonstrated empirically by Peterson et al. across both semantic and perceptual domains, the size principle provides a universal mathematical explanation for how we prevent our generalizations from expanding forever, ensuring our inferences remain tightly anchored to the scale of our experience.
Key Foundational Work
Bayesian Modeling of Human Concept Learning (1998) — contributed 4 concept elements to this direction.
Rules and Similarity in Concept Learning (1999) — contributed 2 concept elements to this direction.