Stating the Obvious: Extracting Visual Common Sense Knowledge

Mark Yatskar, Vicente Ordonez, Ali Farhadi

Abstract

Traditional common sense knowledge bases often suffer from reporting bias, as humans rarely document mundane, "obvious" physical relationships. This paper presents a method to extract "visual common sense" by mining large-scale image datasets. By analyzing object co-occurrences and spatial relationships in MS-COCO and generalizing them via WordNet, the authors uncovered over 400,000 high-quality common sense facts fundamental to understanding the physical world.

Introduction: The Reporting Bias Problem

Artificial Intelligence systems often lack the basic "visual common sense" that humans take for granted. Traditional knowledge bases like ConceptNet or Cyc are primarily built from text mining or manual entry. However, text is subject to reporting bias: people tend to write about the unusual ("man bites dog") rather than the mundane ("bowl holds broccoli").

This lack of visual common sense—knowledge about the spatial and functional properties of entities—creates a gap for AI systems. A robot or a computer vision model needs to know that a person is likely cutting a cake if a knife is touching it, even if a caption doesn't explicitly say so. The authors propose that we should look at what images show rather than just what text says.

Methodology: From Pixels to Knowledge

The researchers developed a three-stage pipeline to extract and expand visual knowledge:

1. Mining Object-Object Relations

Using the Microsoft Common Objects in Context (MS-COCO) dataset, the team analyzed geometric relationships between 80 object categories. They defined six core spatial relations based on pixel overlaps and centroids: touches, above, besides, holds, on, and disconnected.

Definitions of the six core spatial relations used to analyze object interactions.

They used Pointwise Mutual Information (PMI) to rank these relations, identifying statistically significant pairs, such as holds(pizza, broccoli).

2. Mining Entailment Rules

To bridge the gap between geometry and language, the authors paired these spatial relations with natural language captions from MS-COCO. By using skipgrams and a conditional probability model, they derived rules that map physical configurations to linguistic descriptions.

Examples of successful entailment rules mapping geometric relations to natural language interpretations.

3. Generalizing with WordNet

To expand beyond the 80 COCO categories, the authors utilized WordNet’s hierarchy. For instance, if the system knows on(dog, bed), it can generalize to on(domestic animal, furniture). To prevent over-generalization, they formulated this as a submodular k-coverage optimization problem, balancing the breadth of new facts with consistency against the original data.

Key Insights and Results

The authors treated co-occurring objects in images as "typed bigrams," applying NLP statistical tools like PMI to visual scenes. A key finding was the uniqueness of this data: only 4% of their top 100 extracted spatial relations were found in ConceptNet’s AtLocation relations, proving that visual data provides a complementary source of knowledge that text-based mining misses entirely.

Results at a Glance: * Scale: The method extracted over 16,000 initial statements, generalized to over 400,000 potentially useful facts involving unseen objects. * Quality: Human evaluators rated the top quintile of object-object relations above 0.63, significantly higher than the random baseline of 0.22. * Generalization: The system maintained high quality even for objects not present in the training images, such as holds(dining-table, cutlery).

Limitations and Future Work

The generalization model currently assumes that if a relation isn't seen in MS-COCO, it might be false (a "closed world assumption"), which can lead to inaccuracies. Additionally, the study relied on a limited vocabulary of six geometric relations. Future work could extend this to video datasets to learn about temporal common sense and more nuanced interactions like "leaning against" or "tucked under."

Summary updated

Abstract

Obtaining common sense knowledge using current information extraction techniques is extremely challenging. In this work, we instead propose to derive simple common sense statements from fully annotated object detection corpora such as the Microsoft Common Objects in Context dataset. We show that many thousands of common sense facts can be extracted from such corpora at high quality. Furthermore, using WordNet and a novel submodular k-coverage formulation, we are able to generalize our initial set of common sense assertions to unseen objects and uncover over 400k potentially useful facts.
BibTeX
@inproceedings{yatskar2016stating,
  author = {Mark Yatskar and Vicente Ordonez and Ali Farhadi},
  title = {{Stating the Obvious: Extracting Visual Common Sense Knowledge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-1023},
}
Figure