Multimodal News Article Analysis

Arnau RamisaIJCAI 2017 · DOI

TL;DR

This paper introduces the BreakingNews dataset, a large collection of multimodal news articles, and proposes a framework for analyzing them. They demonstrate how combining text and image information can improve tasks like source detection, popularity prediction, and geolocation, highlighting the potential of multimodal deep learning for understanding news content.

What Problem Does This Paper Address?

The paper tackles the challenge of analyzing online news articles, which are increasingly rich in multimodal information. Beyond just text, news articles now include images, videos, tags, and reader comments, all of which offer complementary insights into the same events. Traditional NLP or computer vision approaches often work in isolation, failing to leverage this interconnected data. The authors aim to bridge this gap by developing methods that can jointly process these different modalities to gain a deeper understanding of news content, which is crucial for tasks like content recommendation, trend analysis, and even combating misinformation.

Methods

The authors propose a multimodal deep learning framework that utilizes Convolutional Neural Networks (CNNs) for both text and image processing.

  • Text Representation: They explore two main approaches:
  • TF-IDF Bag-of-Words (BoW): A standard baseline that represents documents as word frequency histograms, weighted by inverse document frequency to boost the importance of discriminative words.
  • Word2Vec Embeddings: They use pre-trained word embeddings, which capture semantic relationships between words, and process them using a small convolutional network. This approach is noted to be more effective.
  • Image Representation: The standard approach of using the final layers (e.g., FC7) of pre-trained deep CNNs (like VGG19 trained on ImageNet) is employed.
  • Multimodal Fusion: The core of their approach involves combining these text and image representations. They explore various loss functions for different tasks, including:
  • Multinomial Logistic Loss: For classification tasks like source detection.
  • Canonical Correlation Analysis (CCA) Loss: To find linear projections that maximize correlations between modalities, useful for tasks where alignment is key.
  • Great Circle Distance (GCD) Loss: For geolocation tasks, measuring the geodesic distance between predicted and ground truth locations.
  • L1 Distance Loss: For regression tasks like popularity prediction.
  • Learning Strategies: They investigate three learning paradigms:
  • Single-task learning: Training a model for each task independently.
  • Multitask learning: Training a single model to perform multiple tasks simultaneously, sharing representations.
  • Transfer learning: Training a model on one task and then fine-tuning it for another.

Datasets

The primary dataset used is the BreakingNews dataset, introduced by the authors. It comprises 100,000 news articles from high-ranked online newspapers published in 2014. The dataset includes:

  • Article text
  • Accompanying images and their captions
  • Geolocation information
  • Reader comments

This dataset is highlighted as an excellent benchmark for data-hungry deep learning models due to its size and multimodal nature.

Key Insights and Ideas

  • Multimodality Matters: Combining text and image information significantly improves performance across various tasks compared to using text alone. This underscores the importance of a holistic approach to news analysis.
  • Transfer Learning Outperforms Multitask Learning: Surprisingly, transfer learning (fine-tuning a model trained on one task for another) generally yielded better results than multitask learning (training a single model for multiple tasks simultaneously). This suggests that forcing shared representations in multitask learning might sometimes hinder performance, especially if tasks are not highly compatible.
  • Image Caption Anonymization Boosts Geolocation: Anonymizing image captions (replacing names, locations, etc.) surprisingly improved image sequence geolocation performance, suggesting that removing explicit textual cues can force the model to rely more on visual context.
  • Word2Vec is a Strong Text Baseline: Using word2vec embeddings processed by a small CNN proved to be a more effective text representation than traditional TF-IDF BoW for their tasks.

Key Results

  • Article Illustration: Using a CNN with concatenated PLACES and VGG19 features, and a W2V matrix for text, achieved a median rank of 405 for ranking relevant images. This improved to 137 when averaging VGG19 features of multiple images per article. Anonymizing captions further reduced the median rank to 42.
  • Task Performance (Table 2):
    • Geolocation: Single-task learning achieved a median Great Circle Distance (GCD) of 0.90. Transfer learning from Source to Geolocation (S→G) achieved 0.92, while Geolocation to Popularity (G→P) achieved 1.16.
  • Popularity: Single-task learning achieved a median L1 distance of 1.09. Transfer learning from Source to Popularity (S→P) achieved the best result with 0.63.
  • Source Detection: Single-task learning achieved a balanced accuracy of 80.7%. Transfer learning from Geolocation to Source (G→S) achieved 80.2%, and from Popularity to Source (P→S) achieved 77.6%.
  • General Trend: In most cases, single-task learning performed best, followed closely by transfer learning. Multitask learning generally underperformed the other two approaches.

Limitations and Future Work

  • Multitask Learning Performance: The authors acknowledge that multitask learning did not perform as well as expected, suggesting that the shared representations might have harmed performance for certain task combinations. They note that further decoupling tasks in CNN architectures did not improve results.
  • Future Work: The paper implicitly suggests that exploring more sophisticated multimodal fusion techniques and investigating task compatibility for multitask learning could be fruitful avenues for future research. The BreakingNews dataset itself provides a rich platform for further exploration of these and other multimodal news analysis tasks.

Summary updated

BibTeX
@inproceedings{ramisa2017multimodal,
  author = {Arnau Ramisa},
  title = {{Multimodal News Article Analysis}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/737},
}
Figure