Bayesian Models of Inductive Generalization

Neville E. Sanjana, Joshua B. TenenbaumNeurIPS 2002 · OpenReview

Introduction

How do we know that if a cow can catch a specific disease, a horse might catch it too? This is the problem of inductive generalization: making broad inferences from sparse data.

For decades, psychologists relied on "similarity-coverage" models. These suggest we generalize based on how similar a new instance is to the examples we've seen, or how well those examples "cover" the broader category. However, these heuristic models often fail to capture the nuances of human reasoning. For instance, why might seeing three sick chimps make us less likely to generalize a disease to a horse than seeing just one sick chimp?

This paper argues that human inductive reasoning is best explained by a Bayesian framework rather than simple similarity-matching. By combining unsupervised hierarchical clustering with a "Bayesian Occam’s Razor," the authors demonstrate a model that predicts human judgment more accurately than traditional psychological theories, treating generalization as a rational statistical inference problem.

Methods: Building a Rational Model

The authors propose a Bayesian model built on two innovative pillars:

1. Unsupervised Hypothesis Spaces Instead of pre-defining what categories a learner might consider, the authors used hierarchical clustering on human similarity judgments of 10 mammals. This created a "taxonomic hypothesis space" where each node in the tree represents a potential concept or property.

Hierarchical clustering of mammals used to generate the hypothesis space.

2. The Bayesian Occam’s Razor To decide which hypothesis is most likely, the model balances two factors:

  • The Prior: A preference for simplicity. It assumes properties are more likely to belong to a few large clusters rather than many disjointed ones.
  • The Likelihood (The Size Principle): This assumes that examples are sampled at random from the true extension of a concept. Therefore, smaller, more specific hypotheses that "tightly fit" the observed data are mathematically more likely than large, vague ones.

By averaging the predictions of all possible hypotheses (weighted by their posterior probability), the model calculates the "strength" of an inductive argument.

Datasets

The study evaluates this model using three primary data sources:

  • Specific Task (Osherson et al., 1990): Human rankings of 36 arguments where premises are pairs of mammals and the conclusion is always "horses" (e.g., {Chimp, Squirrel} $\rightarrow$ Horse).
  • General Task (Osherson et al., 1990): Human rankings of 45 arguments where premises are triplets of mammals and the conclusion is "all mammals."
  • New Behavioral Study: A novel experiment with 24 subjects testing "varying example set composition." This included cases with one example versus three examples of the same kind to specifically test the Size Principle.

Key Insights and Results

Superior Correlation with Human Judgment The Bayesian model consistently outperformed traditional "max-similarity" and "sum-similarity" models. For example, in the general induction task, the Bayesian model achieved a correlation of $\rho = 0.94$ with human data, compared to $0.87$ for the best similarity model.

Scatter plots comparing model predictions to human data across different tasks.

The Size Principle and the "Same-Kind" Effect Traditional similarity models don't care if you see one chimp or twenty; the similarity to a horse remains the same. The Bayesian model, however, realizes that if you see multiple chimps with a disease and only chimps, the disease is likely specific to chimps (a "small" hypothesis). In their new experiment, the authors confirmed that humans generalize less to a new species when given three examples of the same species versus just one. The Bayesian model was the only one to qualitatively predict this narrowing of the hypothesis.

Subjective vs. Objective Taxonomy Interestingly, the model performs much better when using a hypothesis space derived from human similarity judgments rather than actual biological DNA taxonomy. Our inductive leaps are guided by our internal "mental maps," even when those maps are scientifically inaccurate.

Limitations and Future Work

While powerful, the model has limitations. It relies on a pre-existing similarity matrix and doesn't explain how humans decide which features (size, hairiness, speed) are relevant for building that initial taxonomy. Furthermore, while the model uses unions of clusters, real-world reasoning might involve much more complex, non-taxonomic structures like food chains or ecological niches.

Finally, this model operates at the "computational level"—showing what the brain is calculating—but doesn't explain how neurons perform these Bayesian updates. The authors suggest that simple heuristics might actually be the brain's algorithmic way of approximating these complex, rational Bayesian calculations.

Summary updated

Abstract

We argue that human inductive generalization is best explained in a Bayesian framework, rather than by traditional models based on simi- larity computations. We go beyond previous work on Bayesian concept learning by introducing an unsupervised method for constructing flex- ible hypothesis spaces, and we propose a version of the Bayesian Oc- cam’s razor that trades off priors and likelihoods to prevent under- or over-generalization in these flexible spaces. We analyze two published data sets on inductive reasoning as well as the results of a new behavioral study that we have carried out.
BibTeX
@inproceedings{sanjana2002bayesian,
  author = {Neville E. Sanjana and Joshua B. Tenenbaum},
  title = {{Bayesian Models of Inductive Generalization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2002},
  url = {https://openreview.net/forum?id=Bkb3E_W_-B},
}
Figure