Abstract: Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., <flower, red>), actions (e.g., <baby, smile>), interactions (e.g., <man, walking, dog>), and positional information (e.g., <vase, on, table>). The collected annotations are in the form of fact-image pairs (e.g.,<man, walking, dog> and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms.
Introduction
Historically, vision-language models have been trained using a "caption-level" approach, where an entire image is associated with a full descriptive sentence. While useful, captions are inherently noisy and contain multiple overlapping concepts, making it difficult for models to learn precise, localized relationships—such as exactly who is doing what to whom, and where they are located.
Previous attempts to learn specific visual concepts relied on predefined ontologies or web image search, which often struggle with rare concepts and lack relational structure (like complex actions or interactions). To achieve a deeper, structured understanding of visual data, models require training data broken down into localized "facts." However, manually annotating hundreds of thousands of bounding boxes with structured relationships is prohibitively expensive. This research bridges that gap by automatically extracting and grounding these facts from existing image-caption datasets.
The Automated Annotation Pipeline
The authors propose a two-step automatic annotation pipeline designed to scale efficiently across massive datasets.
1. Fact Extraction from Captions
The system reads free-form image captions and extracts structured facts. They define second-order facts as <Subject, Predicate> (e.g., <car, red> or <person, jumping>) and third-order facts as <Subject, Predicate, Object> (e.g., <boy, petting, dog>). To accomplish this, the system utilizes an existing tool called Clausie alongside a novel NLP pipeline, SedonaNLP.
SedonaNLP is specifically designed to handle common caption typos, resolve word sense ambiguity, and extract spatial prepositions. Notably, it uses shallow dependency parsing rather than deep parse trees. The authors found that shallow parsing with confidence cutoffs yielded fewer systematic errors and ran 8-9x faster than traditional deep parsers, which often fail on the ungrammatical text found in many image captions.
2. Localization (Mapping & Grounding)
Once facts are extracted, the system maps the nouns (Subjects and Objects) to visual entities using WordNet, MS COCO object categories, and SUN scene categories. Finally, it "grounds" these entities by assigning them to actual bounding boxes in the image. The authors developed specific heuristics for this, such as taking the union of multiple bounding boxes when a caption uses a plural noun (e.g., "people"), or finding the minimum distance between boxes to localize an interaction.
Datasets and Methodology
The pipeline was tested and validated using several key resources:
- MS COCO (2014): Used as the primary source of images, captions, and baseline object bounding boxes.
- Flickr30K Entities: Provided region-to-phrase correspondences, making the grounding step highly effective.
- WordNet & SUN Database: Served as auxiliary lexical and visual ontologies to help map extracted text nouns to concrete visual categories and scenes.
Key Results
- Scale and Efficiency: The pipeline automatically collected over 380,000 visual fact annotations, representing more than 110,000 unique visual facts. The entire process for both massive datasets took less than one day of processing time on standard CPU platforms.
- Quality: Human evaluation via Amazon Mechanical Turk (over 10,000 responses) showed that ~83.1% of the localized facts were correct and had reasonably accurate bounding boxes. This demonstrates that the data is high-quality enough to train robust visual systems.
- Improved Learning: Training on localized, structured facts provides much stronger signals for vision models than full-image to full-caption mapping. Models trained on this fact-level data dramatically outperform standard caption-level models in downstream tasks.
Conclusion and Future Work
While highly successful, the automated nature of the pipeline has limitations. The system is vulnerable to typos in original captions (e.g., a misspelling of "how" as "house" resulting in the fact <house, ski>). Additionally, grounding heuristics can occasionally fail when multiple instances of an object are present but only one is performing the action described.
Future work will focus on improving semantic triple extraction to exploit more complex sentence structures without sacrificing accuracy. The ultimate goal is to use this massive, automatically generated dataset to train advanced visual perception models capable of deep, structured image understanding.
Summary updated