Self-Supervised Multimodal Opinion Summarization

Jinbae Im, Moonki Kim, Hoyeop Lee, Hyunsouk Cho, Sehee Chung

Abstract

Opinion summarization aims to condense numerous user reviews into a concise paragraph. While most existing methods focus solely on text, users often rely on images and metadata to form opinions. We present MultimodalSum, a self-supervised framework that integrates text, images, and metadata without requiring human-annotated labels. By employing a novel "pivot" training strategy and a gated fusion mechanism, MultimodalSum generates summaries that are more factual and coherent than unimodal alternatives.

Introduction: Beyond Text-Only Summarization

Condensing hundreds of reviews into a single summary is a challenging task, primarily due to the lack of "gold standard" human-written summaries for training. Recent advances have utilized self-supervised learning, treating a single review as a "pseudo-summary" for others. However, these models typically ignore non-textual data like food photos or restaurant attributes (e.g., "Price Range," "Outdoor Seating"). MultimodalSum addresses this gap by fusing heterogeneous data types into a unified summarization framework.

Methodology

MultimodalSum is built on the BART architecture and utilizes a three-stage training pipeline to align different data modalities.

The Three-Stage Pipeline

  1. Text Modality Pretraining: A standard BART-based summarizer is trained using only text reviews.
  2. Non-Text "Pivot" Pretraining: To integrate images and metadata, the pretrained text decoder is frozen and used as a "pivot." Separate image (ResNet-based) and table encoders are trained to produce representations that the frozen decoder can translate back into text. This forces non-textual encoders to map their features into the same semantic space as the text encoder.
  3. End-to-End Fusion: The entire system is trained jointly using a gated fusion mechanism. This allows the model to dynamically weight the influence of text, images, and metadata for each generated word.
The three-stage training pipeline: (a) text pretraining, (b) pivot pretraining for non-text modalities using a frozen decoder, and (c) joint training for multiple modalities.

To bridge the gap between training (generating a specific review) and inference (summarizing general opinions), the model incorporates a "rating deviation" feature. During inference, setting this deviation to zero prompts the model to generate a summary reflecting the "average" sentiment.

Key Insights and Results

The model was evaluated on the Yelp and Amazon Product Review datasets, achieving state-of-the-art results. On Yelp, it reached a ROUGE-L score of 19.84 and a BERT-score of 87.7, outperforming previous self-supervised models like Copycat.

Selective Modality Use

A key finding was the model's ability to selectively use different modalities. Heatmap analysis of the gating mechanism revealed that the model leans on the image encoder when describing visual elements (like "food") and on the table encoder for factual grounding (like restaurant names).

Heatmap analysis demonstrating how the gated fusion mechanism selectively attends to different modalities based on the context of the generated token.

Human evaluations further confirmed these findings, with MultimodalSum receiving significantly higher marks for coherence (0.290) and grammaticality (0.367) compared to unimodal baselines. The inclusion of metadata acts as a "fact anchor," reducing the likelihood of model hallucinations.

Conclusion

MultimodalSum demonstrates that integrating non-textual data significantly improves the quality of opinion summaries. The "pivot" strategy provides a robust way to align disparate data types in a self-supervised manner. Future work may explore more granular image-text alignment and the use of advanced vision-language models like CLIP to further enhance summary detail.

Summary updated

BibTeX
@inproceedings{im2021self,
  author = {Jinbae Im and Moonki Kim and Hoyeop Lee and Hyunsouk Cho and Sehee Chung},
  title = {{Self-Supervised Multimodal Opinion Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.33},
}
Figure