Extracting Visual Knowledge from the Web with Multimodal Learning

Dihong Gong, Daisy Zhe WangIJCAI 2017 · DOI

TL;DR

This paper tackles the challenge of automatically extracting visual object knowledge from the vastness of the web. By combining image data with surrounding text, their multimodal learning approach significantly boosts object detection accuracy compared to methods relying solely on visual information, achieving an 8.48% average precision improvement.

What Problem Does This Paper Address?

Building comprehensive visual knowledge bases, like ImageNet, is crucial for advancing computer vision. However, traditional methods like manual annotation are expensive, time-consuming, and don't scale well. While existing automated approaches leverage web data, they often struggle with noisy, incomplete, or redundant information, leading to suboptimal accuracy. This paper aims to overcome these limitations by developing a multimodal learning system that effectively integrates textual context with visual information from raw web pages to extract visual object knowledge.

Methods

The core of this work lies in a multimodal learning algorithm that fuses textual and visual cues. The authors propose learning multimodal embeddings where text phrases and visual concepts are mapped into a common continuous space. This is achieved by adapting the Skip-Gram model, originally used for learning word embeddings, to handle both visual categories and text phrases extracted from web pages.

Specifically, they define a multimodal objective function that encourages embeddings to be close if a text phrase appropriately describes an image category, or if two visual categories frequently co-occur. This function is then transformed into a Skip-Gram-like optimization problem, allowing them to leverage efficient implementations like hierarchical softmax.

For prediction, they employ a logistic regression model that uses these multimodal embeddings to predict the presence of visual objects in an image. This model considers both visual co-detection (objects detected by a visual detector) and text information (tags extracted from the surrounding web page). To further refine predictions, they unify the output of this multimodal model with the confidence scores from a state-of-the-art visual object detector (Fast R-CNN) to produce a final scoring function.

Datasets

The authors utilize a large-scale dataset derived from the Common Crawl dataset. They process billions of raw web pages, extracting images and associated text. For their study, they focus on a subset containing hundreds of millions of webpages. They also mention using ImageNet for initializing visual object detectors.

Key Insights and Ideas

  • Text is a powerful signal for visual understanding: The paper strongly advocates for the idea that textual context surrounding images on the web is not just supplementary but crucial for improving the accuracy of visual object extraction.
  • Bridging the gap between pure visual and multimodal approaches: By demonstrating significant precision gains, the work highlights the limitations of relying solely on visual cues in the often noisy and diverse environment of the web.
  • Adaptability of language models to multimodal tasks: The successful adaptation of the Skip-Gram model, a purely text-based algorithm, to learn joint text-visual embeddings is a key methodological insight.
  • The importance of structure learning for noisy web data: The proposed logistic regression model with L1 regularization is designed to handle the large vocabulary of multimodal words and learn sparse, interpretable relationships, making it robust to noisy web data.

Key Results

The paper reports a significant improvement in extraction precision. On average, across 46 object categories, their multimodal approach achieved an 8.48% increase in precision compared to a unimodal (visual-only) baseline. This translates to a 31% reduction in error rates. For specific categories like "beach," precision improved from 74% to 91%, and for "fish," it jumped from 9% to 29%.

Limitations and Future Work

The authors acknowledge that while their system significantly improves precision, it still relies on the performance of the underlying visual object detector. False positives from the detector can still impact the final results. They also note that the text tagging process can sometimes produce inaccurate or incomplete tags, which can be a challenge for multimodal learning. Future work could explore more robust methods for handling noisy text and further integrating other modalities like video.

Summary updated

BibTeX
@inproceedings{gong2017extracting,
  author = {Dihong Gong and Daisy Zhe Wang},
  title = {{Extracting Visual Knowledge from the Web with Multimodal Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/238},
}
Figure