Bayesian Modeling of Human Concept Learning

Joshua B. TenenbaumNeurIPS 1998 · OpenReview

Humans possess a remarkable ability to learn new concepts from just a few positive examples. A toddler can learn what a "dog" is after seeing only a couple of them, without needing to be shown a cat and explicitly told "this is not a dog." Traditional machine learning, however, often requires massive datasets and explicit negative evidence to define boundaries.

This paper proposes a Bayesian framework centered on the "size principle" to accurately model and explain this human generalization behavior.

The Challenge of Few Positive Examples

The paper tackles the gap between human and machine concept learning by focusing on a specific, analytically tractable task: learning axis-parallel rectangles in a multidimensional feature space.

Imagine "healthy insulin levels" are defined by a specific range on two dimensions (e.g., cholesterol and insulin). How does a person decide the boundaries of that "healthy" rectangle after seeing only three or four data points?

Hypothesis space with few examples

When presented with only a few positive examples, there are many possible rectangles (hypotheses) that could encompass them, making it difficult for standard algorithms to generalize without negative examples to constrain the outer bounds.

Methods: The Size Principle

To understand how humans solve this, the author introduces a principled Bayesian model and compares it against two standard approaches:

  1. MIN Algorithm: A conservative approach that generalizes only to the smallest possible rectangle encompassing all observed points (the "bounding box").
  2. Weak Bayes: A model that assumes examples are generated by a process independent of the concept itself, meaning it doesn't favor smaller hypotheses over larger ones.

The core innovation of the proposed Bayesian model is its likelihood function, which assumes that positive examples are a random sample from the true concept. This leads to the size principle: the probability of observing a set of points $X$ given a hypothesis $h$ is $1/|h|^n$, where $|h|$ is the size (area) of the rectangle and $n$ is the number of examples.

Hypothesis space with many examples

As the number of examples ($n$) increases, smaller hypotheses that fit the data become exponentially more likely than larger ones. If you see many points clustered in a tiny area, it is highly improbable they were sampled from a giant rectangle; it is much more likely they come from a small one.

Experimental Insights

To test this, the author conducted a human subject experiment where participants saw "dots" (positive examples) on a screen and had to draw the rectangle they thought represented the full concept. The experiment varied the spatial range of the dots and the number of dots ($n = 2, 3, 4, 6, 10, 50$).

Key Findings: * Automatic Interpolation: The Bayesian model naturally bridges two regimes of cognition. With few examples, it produces broad "similarity-based" gradients (like Weak Bayes). As examples increase, it converges toward sharp, "rule-based" boundaries (like the MIN algorithm). * Model Comparison: The MIN algorithm failed because it was too conservative, predicting zero generalization beyond the dots. Weak Bayes failed because it couldn't account for the number of examples or their range. The Bayesian Model, however, captured the non-linear dependencies of human behavior perfectly. * The Role of Priors: While an "uninformative prior" works adequately, an "expected-size prior" (which assumes concepts have a typical scale) provides a much better fit for human data. Humans bring real-world expectations about scale to any learning task.

Conclusion

By formalizing the size principle, this Bayesian framework explains why we trust a tight fit more than a loose one when learning from sparse data. While the study focuses on simple axis-parallel rectangles, the underlying principles offer a pathway for scaling up to more complex domains, such as learning linguistic grammars, object categories, and automatically detecting mislabeled data in machine learning systems.

Summary updated

Abstract

I consider the problem of learning concepts from small numbers of pos(cid:173) itive examples, a feat which humans perform routinely but which com(cid:173) puters are rarely capable of. Bridging machine learning and cognitive science perspectives, I present both theoretical analysis and an empirical study with human subjects for the simple task oflearning concepts corre(cid:173) sponding to axis-aligned rectangles in a multidimensional feature space. Existing learning models, when applied to this task, cannot explain how subjects generalize from only a few examples of the concept. I propose a principled Bayesian model based on the assumption that the examples are a random sample from the concept to be learned. The model gives precise fits to human behavior on this simple task and provides qualitati ve insights into more complex, realistic cases of concept learning.
BibTeX
@inproceedings{tenenbaum1998bayesian,
  author = {Joshua B. Tenenbaum},
  title = {{Bayesian Modeling of Human Concept Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1998},
  url = {https://openreview.net/forum?id=r1VVhuZOWr},
}
Figure