Rules and Similarity in Concept Learning

Joshua B. TenenbaumNeurIPS 1999 · OpenReview

Introduction

For decades, cognitive scientists have debated how the human mind learns and generalizes concepts. The traditional view posits a strict dichotomy between two distinct systems: one that abstracts logical rules (e.g., "a prime number") and another that computes similarity to stored examples (e.g., "this looks like the number 60"). Most psychological models treat these as separate, modular systems that somehow interact.

However, these modular accounts struggle to explain why the mind chooses specific rules or similarity metrics, and why our reasoning seamlessly shifts from similarity-like patterns to rule-like patterns as we gain more experience. In this paper, Joshua Tenenbaum demonstrates that this divide is a false dichotomy. By applying a Bayesian framework equipped with a "size principle," he shows that both modes of reasoning emerge naturally from probabilistic inference, depending entirely on the amount and distribution of the observed data.

A Unified Bayesian Framework

The paper introduces a Bayesian framework for concept learning that avoids hardcoding separate rule and similarity modules. The core idea is that a learner entertains a "hypothesis space" ($H$) of possible concepts. When shown positive examples ($X$), the learner updates their belief in each hypothesis using Bayes' rule.

Two key mechanisms drive this unified model:

  1. The Size Principle: This is the mathematical "secret sauce" of the framework. The likelihood of a hypothesis is inversely proportional to its size or extension. Mathematically, $p(X|h) = 1/|h|^n$, where $n$ is the number of examples. If you see the number 16 and the concept is "powers of two" (a small, sparse set), that is a much more likely explanation than "even numbers" (a large, dense set). As more consistent examples are observed, this principle exponentially penalizes larger hypotheses.
  2. Hypothesis Averaging: Instead of just picking the single best rule, the model computes the probability that a new object belongs to the concept by averaging the predictions of all consistent hypotheses, weighted by their posterior probability.

To prove the necessity of these two mechanisms, the author compares the full Bayesian approach against two "lesioned" versions:

  • SIM (Pure Similarity): A model that ignores the size principle, treating all consistent hypotheses as equally likely (binary likelihood).
  • MIN (Pure Rule): A model that picks only the most specific consistent hypothesis (maximum likelihood) instead of averaging across the hypothesis space.

Experimental Evidence and Results

The paper evaluates these models using experimental datasets, most notably the Number Concept Game. Subjects were shown numbers between 1 and 100 and asked to generalize. Trials were split into three classes to test different types of reasoning:

  • Class I: Single examples (e.g., $\{16\}$).
  • Class II: Examples following a mathematical rule (e.g., $\{16, 8, 2, 64\}$).
  • Class III: Examples following a magnitude interval (e.g., $\{16, 23, 19, 20\}$).
Comparison of human generalization judgments against the Bayesian, pure similarity (SIM), and pure rule (MIN) models.

The Bayesian model significantly outperformed the traditional modular competitors in fitting human data. For the Number Concept Game, the Bayesian model achieved an impressive $R^2 = 0.91$, compared to $0.74$ for the similarity-only model (SIM) and a poor $0.47$ for the rule-only model (MIN). In a separate Word Learning task involving taxonomic hierarchies, the Bayesian fit was even stronger at $R^2 = 0.98$.

Qualitatively, the Bayesian model correctly predicted the "all-or-none" generalization for mathematical rules (Class II) and the "graded" generalization for magnitude-based clusters (Class III), all without needing to switch between different cognitive algorithms.

Key Insights

By unifying these phenomena under one mathematical umbrella, the paper yields several profound insights into human cognition:

  • The Emergence of Rules: Rule-like behavior isn't a separate "mode." It happens when the size principle causes the posterior probability to concentrate heavily on a single, highly specific hypothesis. This typically occurs when multiple examples are consistent with a "sparse" rule.
  • The Persistence of Similarity: Similarity-like gradients appear when the posterior is "spread out" over many overlapping hypotheses. This happens when we have very little data (e.g., just one example) or when the data fits "dense" hypotheses like magnitude intervals.
  • The Speed of Convergence: The model explains why we jump to rules quickly in some domains (like math or word taxonomy) but stay in a "similarity mode" longer in others (like continuous physical dimensions). It depends entirely on the structural overlap of the hypotheses in that specific domain.
  • A Principled Approximation: Tenenbaum suggests that while the brain might actually implement something akin to SIM or MIN for cognitive efficiency, the full Bayesian model provides the "computational-level" explanation for why those approximations are useful in the first place.

Limitations and Future Work

While powerful, the framework has limitations that invite future research. First, the model assumes the hypothesis space is already given—constructed via prior knowledge or clustering. It does not explain how humans generate or discover these hypotheses from scratch in a novel environment. Second, while the paper aligns with behavioral data, mapping the Bayesian "averaging" and "size principle" onto specific neural circuits remains an open challenge. Finally, the experiments involve relatively small, discrete domains; scaling this exact probabilistic inference to high-dimensional, real-world sensory data remains a significant computational hurdle.

Summary updated

Abstract

This paper argues that two apparently distinct modes of generalizing con(cid:173) cepts - abstracting rules and computing similarity to exemplars - should both be seen as special cases of a more general Bayesian learning frame(cid:173) work. Bayes explains the specific workings of these two modes - which rules are abstracted, how similarity is measured - as well as why gener(cid:173) alization should appear rule- or similarity-based in different situations. This analysis also suggests why the rules/similarity distinction, even if not computationally fundamental, may still be useful at the algorithmic level as part of a principled approximation to fully Bayesian learning.
BibTeX
@inproceedings{tenenbaum1999rules,
  author = {Joshua B. Tenenbaum},
  title = {{Rules and Similarity in Concept Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
  url = {https://openreview.net/forum?id=r1N_jubuZS},
}
Figure