Deep Submodular Optimization and LLM for Multimodal Content Extraction and Automatic Poster Generation from Long Document

Vijay Jaisankar, Sambaran Bandyopadhyay, Kalp Vyas, Varre Suman Chaitanya, Shwetha SomasundaramAAAI 2025 · DOI

Introduction

Creating a scientific poster from a long research document is a high-cognitive-load task that requires a sophisticated multimodal summary. A successful poster must balance three competing requirements: coverage (representing the entire document), diversity (avoiding redundant information), and alignment (ensuring visual elements correspond to the surrounding text).

While Large Language Models (LLMs) excel at text summarization, they often struggle with long-context multimodal documents, frequently exhibiting hallucinations or losing track of information in lengthy files. Furthermore, LLMs typically lack the "design sense" necessary for selecting appropriate fonts, colors, and layouts. Existing automated tools often rely on rigid templates or require significant manual intervention. To address these challenges, we introduce PostDoc, an end-to-end pipeline designed to transform long documents into visually appealing one-page posters efficiently.

The PostDoc Pipeline

PostDoc utilizes a two-stage approach to automate poster generation, combining mathematical optimization with the generative power of LLMs.

1. Content Extraction via Deep Submodular Optimization

Rather than processing the entire document through an LLM, PostDoc employs a trainable "deep submodular" optimization function for content selection. Submodular functions are characterized by the law of diminishing returns—the marginal value of adding information decreases as the summary grows.

The PostDoc pipeline architecture, illustrating the flow from input document to generated poster.

The innovation lies in the design of a trainable submodular function that explicitly rewards coverage, diversity, and image-text alignment. By training on ground-truth summaries, the model learns to identify "poster-worthy" segments. A critical component is the Multimodal Alignment term, which ensures that if an image is selected, the system prioritizes the text that explains it.

2. Paraphrasing and Design Synthesis

Once the core content is extracted, it is refined through several specialized modules:

  • Refinement: Extracted text is sent to GPT-3.5-turbo to be rewritten into concise, poster-friendly bullet points.
  • Typography & Color: A fine-tuned MiniLM selects fonts based on the title, while a specialized model (TPNSmall) generates a cohesive color palette.
  • Layout Generation: A heuristic-based approach automatically arranges images and text, dynamically adjusting box heights to accommodate the content.

Experimental Evaluation

The system was evaluated using three primary datasets: MSMO (multimodal summarization), NJU-Fudan (scientific papers and posters), and Let Me Choose (font pairings).

Quantitative Results

PostDoc demonstrated significant improvements in both quality and efficiency. On the MSMO dataset, it achieved a ROUGE-L score of 0.68, nearly doubling the performance of high-end baselines like BRIO (0.36).

Performance comparison of PostDoc against baseline methods across MSMO and NJU-Fudan datasets.

Crucially, PostDoc is approximately 20x faster than standard LLM-only approaches. The inference time per document was recorded at 0.68s, compared to 14.88s for a GPT-3.5+BLIP baseline. This efficiency stems from using submodular optimization to filter the document before LLM processing, which also reduces computational costs.

Human Evaluation and Generalization

In a blind study with AI experts, PostDoc received significantly higher ratings for its Template (3.46/5) and Coverage (3.40/5) than LLM-based baselines. Furthermore, the system showed strong zero-shot performance on the NJU-Fudan dataset, proving its ability to generalize to diverse academic document types without specific training on those formats.

Conclusion and Future Work

PostDoc provides a robust framework for automated multimodal summarization and design. While highly effective, the system currently faces limitations with complex visual data like flowcharts or neural network diagrams, which it treats as standard images rather than structural data. Future research will focus on fine-tuning Vision Language Models (VLMs) to better interpret document structures and increasing layout flexibility to match the creative variety of human designers.

Summary updated

BibTeX
@inproceedings{jaisankar2025deep,
  author = {Vijay Jaisankar and Sambaran Bandyopadhyay and Kalp Vyas and Varre Suman Chaitanya and Shwetha Somasundaram},
  title = {{Deep Submodular Optimization and LLM for Multimodal Content Extraction and Automatic Poster Generation from Long Document}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34598},
}
Figure