Cross-Modal Commentator: Automatic Machine Commenting Based on Cross-Modal Information

Pengcheng Yang, Zhihan Zhang, Fuli Luo, Lei Li, Chengyang Huang, Xu Sun

Introduction

Automatic commenting systems have traditionally been "blind," relying solely on the textual content of news articles. However, real-world news is inherently multi-modal, often featuring rich imagery that provides context or aesthetic details not explicitly mentioned in the text. Researchers from Peking University and BUPT have introduced Cross-Modal Automatic Commenting (CMAC), a task where AI generates comments by integrating both textual and visual information.

This approach addresses the "information gap" where human readers often comment on visual cues—such as the lack of green grass in a photo of peach blossoms—that a text-only model would miss. By bridging the gap between pixels and words, the model produces comments that are significantly more diverse and relevant.

An example of the CMAC task, showing how visual information (the lack of green grass) influences human comments in ways text alone cannot.

Methodology: The Co-Attention Model

The proposed solution is a Co-Attention Model designed to capture the intrinsic interactions between modalities. The architecture comprises three primary components:

  1. Encoders: Textual features are extracted using Gated Recurrent Units (GRU) or Transformers from the news title and body. Visual features are extracted from images using a pre-trained ResNet-152 model.
  2. Co-Attention Mechanism: This core innovation calculates a similarity matrix between text and images. It generates vision-aware textual representations (words related to images) and text-aware visual representations (image regions related to the narrative).
  3. Decoder: A GRU or Transformer-based decoder generates the final comment by attending to four sources: raw text, raw images, vision-aware text, and text-aware images.

The CMAC Dataset

To support this research, the authors curated a large-scale benchmark dataset from Netease News. The CMAC Dataset contains 24,134 news articles and 930,656 high-quality human comments. Each entry includes a title, body, and an average of 5.8 images. Comments were filtered to a length of 5–30 characters to ensure meaningful content and meaningful interactions between the text and visual elements.

Distribution of comment lengths (number of characters) in the CMAC dataset.

Key Insights and Results

The research highlights several critical findings regarding the synergy of multi-modal data:

  • Mutual Boosting: Modalities are symbiotic; text guides visual attention, while images highlight significant textual keywords.
  • Diversity and the "Red Word" Phenomenon: Text-only models often produce generic, "safe" responses. By incorporating visual cues, the CMAC model captures "red words"—information present in images but absent from text—leading to more specific and diverse comments.
  • Performance: The Transformer-based CMAC model achieved a 7.7 BLEU-1 score, a 13% improvement over simple text+image concatenation baselines. Human evaluations also rated the model significantly higher in informativeness (5.0/10) and relevance (5.9/10) compared to existing systems.

Conclusion

The Cross-Modal Commentator moves AI closer to human-like interaction by acknowledging that news is a graphic medium. While challenges remain in understanding complex visual scenes and capturing emotional nuances, the co-attention mechanism provides a robust framework for more intelligent, multi-modal content generation.

Summary updated

Abstract

Automatic commenting of online articles can provide additional opinions and facts to the reader, which improves user experience and engagement on social media platforms. Previous work focuses on automatic commenting based solely on textual content. However, in real-scenarios, online articles usually contain multiple modal contents. For instance, graphic news contains plenty of images in addition to text. Contents other than text are also vital because they are not only more attractive to the reader but also may provide critical information. To remedy this, we propose a new task: cross-model automatic commenting (CMAC), which aims to make comments by integrating multiple modal contents. We construct a large-scale dataset for this task and explore several representative methods. Going a step further, an effective co-attention model is presented to capture the dependency between textual and visual information. Evaluation results show that our proposed model can achieve better performance than competitive baselines.
BibTeX
@inproceedings{yang2019cross,
  author = {Pengcheng Yang and Zhihan Zhang and Fuli Luo and Lei Li and Chengyang Huang and Xu Sun},
  title = {{Cross-Modal Commentator: Automatic Machine Commenting Based on Cross-Modal Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1257},
}
Figure