Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution

Abhilash Nandy, Sambaran BandyopadhyayAAAI 2025 · DOI

TL;DR

This paper introduces MiDAS-PRo, a novel framework for multi-document summarization that addresses the critical need for attribution (citing sources). It cleverly leverages large language models (LLMs) trained on code to act as few-shot planners and reasoners, generating hierarchical summaries with source attribution without requiring extensive fine-tuning.

What Problem Does This Paper Address?

Summarizing information from multiple documents is a common and increasingly important task, especially with the explosion of digital content. However, simply concatenating documents and summarizing them often leads to redundant, poorly structured, and untrustworthy outputs. A key challenge is ensuring that each piece of information in the summary can be traced back to its original source – a process known as attribution. Existing methods often struggle with this, either lacking attribution capabilities or requiring expensive fine-tuning and specialized datasets. This paper tackles the problem of generating coherent, multi-document summaries that are not only informative but also transparent and reliable due to accurate source attribution, all while minimizing the need for extensive training data.

Methods

MiDAS-PRo employs a three-stage approach:

  1. Planning (Hierarchical Reference Layout Tree - HRLT Generation): This stage uses code-LLMs to predict the hierarchical structure of how references will appear in the final summary. It determines which citations should be grouped together and which should appear in the same sentence, creating a "plan" for the summary's organization. This is framed as a code completion task.
  2. Reasoning (Sentence-Wise Entity Generation): Here, code-LLMs are used again to identify key entities or topics relevant to each sentence that will contain citations. This acts as a "reasoning scaffold" to ensure the summary's content is well-supported and coherent. This is also treated as a code completion task.
  3. Summary Generation: The generated planning and reasoning information is then combined with the original documents and intent into a natural language prompt for a standard LLM to produce the final attributed summary.

A crucial component is the GAT-based In-Context Example Selection. To ensure the few-shot examples provided to the LLM are highly relevant, a Graph Attention Network (GAT) is trained to learn embeddings that represent document collections. This GAT helps select the most similar examples from a training set, minimizing the need for manual curation.

Datasets

The paper uses two primary datasets:

  • Multi-XScience: A large-scale dataset of scientific articles, where the task is to generate a "related work" section for a query paper, with citations to source abstracts.
  • MiDAS: A subset of the BigSurvey-MDS dataset, specifically curated to include reference attribution information. Here, the task is to generate the introduction paragraph of a survey paper, citing its references.

The authors also introduce MiDAS, a manually curated subset of BigSurvey-MDS with added reference attribution, to evaluate their approach.

Key Insights and Ideas

  • Code LLMs as Few-Shot Planners/Reasoners: The paper demonstrates that LLMs trained on code can effectively perform planning and reasoning tasks for complex NLP problems like multi-document summarization, even in a few-shot setting. This is a significant finding, as it opens up new avenues for leveraging code-based models beyond traditional code generation.
  • Hierarchical Planning for Attribution: The HRLT generation is key to ensuring logical flow and correct attribution. By planning the structure of citations, the model can create more coherent and interpretable summaries.
  • GAT for Relevant In-Context Examples: Using a GAT to select in-context examples is a novel and effective way to improve few-shot learning performance. It highlights the importance of carefully choosing examples for LLMs.
  • Code-based Reasoning Enhances Interpretability: The use of code generation for planning and reasoning makes the LLM's decision-making process more transparent, contributing to the interpretability of the generated summaries.

Key Results

  • MiDAS-PRo consistently outperforms various baselines across multiple LLMs (Mistral-7B-Instruct, Llama-3-8B-Instruct, Llama-3-70B-Instruct, GPT-40) on both summarization (ROUGE-L, BLEU, BERTScore) and clustering-based metrics (NMI, AMI) for attribution evaluation on the Multi-XScience dataset.
  • Significant improvements are observed in clustering-based metrics, with percentage improvements in average NMI and AMI scores often exceeding 20% and even reaching over 100% in some cases.
  • MiDAS-PRo achieves the best semantic performance (BERTScore) when applied to GPT-40.
  • Human evaluations show MiDAS-PRo scoring highest in relevance, fluency, attribution correctness, and overall score compared to baselines.
  • The paper also shows that increasing model size doesn't always guarantee better performance for this specific task, suggesting the difficulty of multi-document summarization for scientific articles.

Limitations and Future Work

  • Dataset Size for MiDAS: The MiDAS dataset, while curated, is relatively small (40 training, 56 test examples), which might limit its generalizability.
  • Reliance on Code LLMs: While effective, the reliance on code LLMs for planning and reasoning might be a bottleneck if such models are not readily available or performant for specific domains.
  • Future Work: The authors suggest exploring more sophisticated GAT architectures for ICE selection and investigating the application of MiDAS-PRo to other domains beyond scientific articles. They also highlight the potential for further improvements by exploring different LLMs and fine-tuning strategies.

Summary updated

BibTeX
@inproceedings{nandy2025language,
  author = {Abhilash Nandy and Sambaran Bandyopadhyay},
  title = {{Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34676},
}
Figure