LAMS: A Location-aware Approach for Multimodal Summarization (Student Abstract)

Zhengkun Zhang, Jun Wang, Zhe Sun, Zhenglu YangAAAI 2021 · DOI

Abstract

Multimodal summarization (MMS) typically focuses on the semantic fusion of text and images. However, the structural intent behind image placement—often a strong signal of sentence importance—is frequently overlooked. We present LAMS (Location-aware Multimodal Summarization), a framework that treats image location as a critical feature for identifying salient information. By utilizing a Transformer-based architecture with a specialized Multimodal Fusion Block (MFB) and a Pointer Network decoder, LAMS achieves state-of-the-art results on the MSMO dataset. Our findings demonstrate that "where" an image is placed provides a meaningful boost to summarization quality, acting as a proxy for human attention.

Introduction

In multimodal summarization, the primary challenge is effectively integrating visual features with textual data. Traditional models assume that the semantic content of an image clarifies the surrounding text. However, this ignores the author's structural intent. Image placement in an article is rarely random; it typically anchors or highlights the most significant nearby sentences. Existing models often devalue image locations because they are seen as difficult to quantify. LAMS addresses this by formalizing image location as a structural signal of importance, moving beyond simple content-based fusion.

Methodology

The LAMS architecture consists of three primary modules: Input Representation, an Encoder Module featuring a Multimodal Fusion Block (MFB), and a Decoder Module.

The LAMS architecture overview, showcasing the Input Representation, Encoder with MFB, and the Pointer Network-based Decoder.

Input Representation

Textual input is encoded using BERT followed by a convolutional layer to capture local context. Visual features are extracted from images using a pre-trained VGG19 network, followed by a linear layer to align feature dimensions.

Multimodal Fusion Block (MFB)

The MFB is the core innovation of LAMS. Instead of simple concatenation, it employs bilinear pooling to capture high-order interactions between text and images. These blocks are stacked to model the "maximum influence range" of an image, allowing the model to weight sentence importance based on proximity to visual elements.

Detailed view of the Multimodal Fusion Block (MFB) architecture.

Summary Decoder

LAMS utilizes a Pointer Network as its decoder, making it an extractive summarization tool. The network identifies and extracts the most salient sentences from the source document to construct the final summary, rather than generating new text from scratch.

Dataset and Experiments

The model was evaluated on an extended version of the MSMO (Multimodal Summarization with Multimodal Output) dataset. The original dataset lacked image location data, which the authors recovered by crawling the original Daily Mail URLs. The final dataset comprises 287,467 training samples, 10,265 validation samples, and 10,261 testing samples.

LAMS was compared against several baselines, including text-only models (Lead-3, Pointer-Generator) and existing multimodal models (HAN, ATL).

Results

LAMS demonstrated superior performance across all standard ROUGE metrics. Specifically, it achieved a ROUGE-1 score of 43.07, outperforming both text-only and content-only multimodal baselines. These results were statistically significant, proving that adding "where" an image is to the "what" of an image provides a meaningful boost to summarization quality.

Metric LAMS Score
ROUGE-1 43.07
ROUGE-2 20.28
ROUGE-L 39.34

Conclusion

LAMS proves that structural cues, such as image placement, are essential for high-quality multimodal summarization. By treating location as a proxy for human attention, the model identifies salient information more effectively than content-only approaches. Future work will explore applying these location-aware insights to abstractive summarization models to generate more fluid and cohesive summaries.

Summary updated

BibTeX
@inproceedings{zhang2021lams,
  author = {Zhengkun Zhang and Jun Wang and Zhe Sun and Zhenglu Yang},
  title = {{LAMS: A Location-aware Approach for Multimodal Summarization (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17971},
}
Figure