Introduction
Query-focused summarization is the task of extracting sentences from a document that best answer a specific user query. This task is notoriously difficult because queries are often very short—typically only 3-4 words. This "paucity of information" makes it challenging for standard algorithms to distinguish between a sentence that is generally important to a document and one that is specifically relevant to the user's information need.
While the Information Retrieval (IR) community often employs ad-hoc "query expansion" heuristics to address this, there is a need for a statistically justified framework. The authors introduce BAYESUM (Bayesian Summarization), a generative model that treats query-focused summarization as a principled form of query expansion by leveraging multiple documents known to be relevant to a single query.
The BAYESUM Model
BAYESUM is a hierarchical Bayesian generative model based on language modeling principles. The core intuition is that every word in a document is generated from a mixture of three distinct sources:
- General English ($p^G$): Common "filler" words.
- Document Background ($p^{d_k}$): Words specific to a particular document but not the query.
- Query-Specific ($p^{q_j}$): Words that define why a document is relevant to a specific query.
The model operates at two levels. At the word level, a hidden variable ($z$) determines which of the three sources generated a specific word. At the sentence level, a hidden variable ($\pi$) represents the degree to which an entire sentence belongs to each source. Because exact Bayesian inference is intractable, the authors utilize Expectation Propagation (EP), a deterministic approximation technique that is faster and more accurate for these mixture models than traditional variational methods. To generate a summary, the model estimates the query-specific language model and ranks sentences based on their KL-divergence to that model.
Experimental Evaluation
The model was evaluated using several standard benchmarks, including the TREC (Text REtrieval Conference) dataset for large-scale training and a human-annotated gold standard of 166 query/document pairs. Additionally, the model was tested on multi-document summarization tasks from MSE 2005 and DUC 2005.
Key Results
BAYESUM significantly outperformed state-of-the-art baselines. On the TREC dataset, it achieved a Mean Average Precision (MAP) of 44.1, compared to 36.6 for the standard KL-divergence IR baseline and 24.8 for a position-based baseline.
In international evaluations, BAYESUM ranked 1st in human "pyramid" evaluations at MSE 2005 and achieved the highest responsiveness score among approximately 30 participating systems at DUC 2005. Despite the complexity of Bayesian inference, the EP algorithm allowed the model to process the massive TREC dataset in just 2.5 hours.
Insights and Robustness
A key insight from this work is that BAYESUM effectively "learns" the meaning of a query by observing all documents relevant to it. Remarkably, the model remains highly effective even when the query text is completely removed, as it can infer the "hidden" query intent from the document cluster. Furthermore, the model demonstrates high robustness to noise; it successfully filters out background information even when the initial document retrieval is imperfect.
Conclusion and Future Work
While BAYESUM represents a significant step forward in extractive summarization, it currently lacks a mechanism to generate original sentences (abstractive summarization) or to internally manage redundancy in multi-document settings. Future work may involve moving beyond the "bag-of-words" assumption by incorporating n-gram models or dependency trees to better capture syntax and word order.
Summary updated