News Content Completion with Location-Aware Image Selection

Zhengkun Zhang, Jun Wang, Adam Jatowt, Zhe Sun, Shao-Ping Lu, Zhenglu YangAAAI 2021 · DOI

Abstract

Automated journalism systems often struggle with the "where-which" problem: determining the optimal locations within a long-form article to insert images and selecting the most contextually relevant visuals for those slots. We present a two-stage framework that addresses these challenges through a hierarchical text encoding approach and multimodal fusion. By utilizing a sentence-level Image Location Extractor and a context-aware Image Selector, our model effectively manages local and global information to produce visually coherent news content. Experimental results on the NYTimes800k and MSMO datasets demonstrate that our framework significantly outperforms existing state-of-the-art methods in both location prediction and image retrieval accuracy.

Introduction

While AI has made significant strides in text generation and single-image retrieval, real-world news articles are linguistically complex and often cover multiple sub-events. A single representative image is frequently insufficient to capture the full narrative. This research identifies and tackles the "where-which" problem:

  1. Where: Identifying specific paragraphs or sentences within a long article that are salient enough to warrant visual support.
  2. Which: Selecting the best image from a gallery that represents the local context of a specific slot while maintaining consistency with the article's overall theme.

Existing models often rely on simple captions and fail to handle the redundancy and diversity requirements of multi-image selection. Our approach extracts key information from long-form news to select a series of images that are both locally relevant and globally diverse.

Methodology: A Two-Stage Framework

Our proposed end-to-end framework consists of two primary components designed to handle the hierarchical nature of news content.

Stage 1: Image Location Extractor

To capture the structure of news, the model encodes text at the sentence level using RoBERTa and a CNN. These embeddings are processed by a Transformer to capture global context. A Pointer Network then identifies the best insertion locations sequentially, treating the article as a series of potential slots.

Stage 2: Image Selector

For each chosen location, the model generates a "Context Embedding" by merging local slot information with global article data. We employ Multimodal Tucker Fusion (MUTAN) to model the interactions between text and candidate images, calculating similarity scores to select the most appropriate visual.

The proposed two-stage architecture featuring the Image Location Extractor and the Image Selector with Multimodal Tucker Fusion.

Sequential Insertion Strategies

To ensure a "human-like" selection process, we introduced three key strategies:

  • Coverage Mechanism: Prevents image clustering by tracking previous insertion points.
  • Two-stream Similarity: Penalizes the selection of redundant images to ensure visual diversity.
  • Image Pointer: Feeds information from previously selected images back into the model to maintain narrative coherence.

Experimental Results and Insights

The model was evaluated on two large-scale datasets: NYTimes800k (444,000+ articles) and an extended version of MSMO (307,000+ articles).

Quantitative Performance

Our framework consistently outperformed baselines like sm-LSTM and Newsroom. On the NYTimes800k dataset, the model achieved a 98.8% Recall@1 for predicting the initial image location. In the more complex task of selecting a series of correct images (R/N), the full model achieved scores of 11.9 on NYT and 9.9 on MSMO, surpassing all previous state-of-the-art methods.

Qualitative Success

Qualitative analysis shows the model's ability to distinguish between different sub-events within a single story. For example, in an article about a football manager's departure, the model successfully identified distinct segments—such as the manager's early history, a specific match result, and a career summary—and matched them with unique, relevant photos.

Qualitative example showing the model identifying distinct sub-events (highlighted in different colors) within a news article to determine optimal image placement.

Conclusion

This research demonstrates that for multi-image selection in news, local context is paramount. By balancing local and global embeddings and explicitly enforcing visual diversity through mathematical strategies, our framework creates a high-quality, visually enriched reading experience. Future work will explore more complex semantic relationships, such as chronological progression, and the integration of generative models to synthesize custom images when suitable matches are unavailable in existing galleries.

Summary updated

BibTeX
@inproceedings{zhang2021news,
  author = {Zhengkun Zhang and Jun Wang and Adam Jatowt and Zhe Sun and Shao-Ping Lu and Zhenglu Yang},
  title = {{News Content Completion with Location-Aware Image Selection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17704},
}
Figure