AnalogySpace: Reducing the Dimensionality of Common Sense Knowledge

Robyn Speer, Catherine Havasi, Henry LiebermanAAAI 2008

Abstract

AnalogySpace applies Singular Value Decomposition (SVD) to ConceptNet, a massive database of common sense assertions, to create a multi-dimensional vector space of human knowledge. By treating common sense as a matrix of concepts and features, the system can "smooth" noisy data and perform powerful analogical reasoning—such as predicting that a cat has four legs because it is semantically similar to a dog—using simple linear algebra.

Introduction: The Challenge of Common Sense

Common sense is the "dark matter" of AI: it is vast, often unstated, and frequently inconsistent. Traditional symbolic logic struggles with common sense because it requires absolute truths, whereas human knowledge is built on tendencies, connotations, and "fuzzy" similarities.

Previous attempts to automate common sense reasoning, such as the "Learner" system, used a technique called cumulative analogy. However, these rule-based systems are brittle; they fail if exact features do not match perfectly and become computationally expensive as the knowledge base grows. The authors sought a way to make these analogies more robust, scalable, and resistant to the "noise" inherent in crowd-sourced data.

Methodology

The researchers utilized data from the Open Mind Common Sense (OMCS) project, specifically ConceptNet, a semantic network containing over 250,000 assertions contributed by thousands of volunteers. To process this data, they treat common sense as a giant, sparse matrix where rows represent concepts (e.g., "dog") and columns represent features (e.g., "is a pet").

By applying Truncated Singular Value Decomposition (SVD) to this matrix, they create a reduced-dimensional space (typically 50–100 dimensions). In this space, both concepts and features are represented as vectors. Reasoning becomes a matter of vector arithmetic: to find out if a concept should have a certain feature, one calculates the dot product. If two concepts are close together in this space, they are "analogous," and the system can automatically project features from one to the other.

Key Insights: Eigenconcepts and Semantic Smoothing

SVD acts as a noise filter. Idiosyncratic or "bad" assertions that do not correlate with the rest of the data are effectively discarded, while broad, meaningful patterns are amplified. The principal components (the axes of the new space) often represent recognizable human distinctions, termed "Eigenconcepts."

Visualization of AnalogySpace axes representing 'Desirability' and 'Feasibility'.

As seen in the visualization above, the first component in their model distinguished "things people want" (love, money) from "things people don't want" (slavery, losing keys). Other axes represented "feasibility" (walking vs. flying) or "indoors vs. outdoors." Because the space is continuous, the system can also perform ad-hoc categorization by identifying regions (e.g., a "utensil" region) based on the proximity of related vectors.

Key Results

In human trials involving 40 evaluators, AnalogySpace significantly outperformed previous state-of-the-art methods for common sense inference. On a scale where 2 is "Generally true" and -1 is "Nonsense," AnalogySpace inferences scored 1.025, compared to 1.315 for human-entered facts.

Breakdown of user evaluation ratings across different inference methods.

As illustrated in the breakdown of ratings, AnalogySpace produced far fewer "untrue or nonsensical" assertions than the "within-relations" baseline. This proves that analyzing the entire structure of knowledge simultaneously is more effective than looking at individual relation types in isolation.

Limitations and Future Work

Despite its success, the system faces sparsity constraints; the authors had to exclude rare concepts with fewer than four assertions to maintain data density. Additionally, while SVD is efficient, scaling to millions of facts requires significant memory and optimized computational libraries. Future work suggests using AnalogySpace to "clean" ConceptNet by flagging dubious assertions and using it to "bootstrap" new knowledge by asking users to confirm the system's high-confidence inferences.

Summary updated

BibTeX
@inproceedings{speer2008analogyspace,
  author = {Robyn Speer and Catherine Havasi and Henry Lieberman},
  title = {{AnalogySpace: Reducing the Dimensionality of Common Sense Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
}
Figure