Abstract
Traditional common sense knowledge bases often suffer from reporting bias, as humans rarely document mundane, "obvious" physical relationships. This paper presents a method to extract "visual common sense" by mining large-scale image datasets. By analyzing object co-occurrences and spatial relationships in MS-COCO and generalizing them via WordNet, the authors uncovered over 400,000 high-quality common sense facts fundamental to understanding the physical world.
Introduction: The Reporting Bias Problem
Artificial Intelligence systems often lack the basic "visual common sense" that humans take for granted. Traditional knowledge bases like ConceptNet or Cyc are primarily built from text mining or manual entry. However, text is subject to reporting bias: people tend to write about the unusual ("man bites dog") rather than the mundane ("bowl holds broccoli").
This lack of visual common sense—knowledge about the spatial and functional properties of entities—creates a gap for AI systems. A robot or a computer vision model needs to know that a person is likely cutting a cake if a knife is touching it, even if a caption doesn't explicitly say so. The authors propose that we should look at what images show rather than just what text says.
Methodology: From Pixels to Knowledge
The researchers developed a three-stage pipeline to extract and expand visual knowledge:
1. Mining Object-Object Relations
Using the Microsoft Common Objects in Context (MS-COCO) dataset, the team analyzed geometric relationships between 80 object categories. They defined six core spatial relations based on pixel overlaps and centroids: touches, above, besides, holds, on, and disconnected.
They used Pointwise Mutual Information (PMI) to rank these relations, identifying statistically significant pairs, such as holds(pizza, broccoli).
2. Mining Entailment Rules
To bridge the gap between geometry and language, the authors paired these spatial relations with natural language captions from MS-COCO. By using skipgrams and a conditional probability model, they derived rules that map physical configurations to linguistic descriptions.
3. Generalizing with WordNet
To expand beyond the 80 COCO categories, the authors utilized WordNet’s hierarchy. For instance, if the system knows on(dog, bed), it can generalize to on(domestic animal, furniture). To prevent over-generalization, they formulated this as a submodular k-coverage optimization problem, balancing the breadth of new facts with consistency against the original data.
Key Insights and Results
The authors treated co-occurring objects in images as "typed bigrams," applying NLP statistical tools like PMI to visual scenes. A key finding was the uniqueness of this data: only 4% of their top 100 extracted spatial relations were found in ConceptNet’s AtLocation relations, proving that visual data provides a complementary source of knowledge that text-based mining misses entirely.
Results at a Glance:
* Scale: The method extracted over 16,000 initial statements, generalized to over 400,000 potentially useful facts involving unseen objects.
* Quality: Human evaluators rated the top quintile of object-object relations above 0.63, significantly higher than the random baseline of 0.22.
* Generalization: The system maintained high quality even for objects not present in the training images, such as holds(dining-table, cutlery).
Limitations and Future Work
The generalization model currently assumes that if a relation isn't seen in MS-COCO, it might be false (a "closed world assumption"), which can lead to inaccuracies. Additionally, the study relied on a limited vocabulary of six geometric relations. Future work could extend this to video datasets to learn about temporal common sense and more nuanced interactions like "leaning against" or "tucked under."
Summary updated