Concise comparative summaries (CCS) of large text corpora with a human experiment

Jinzhu Jia, Luke Miratrix, Bin Yu, Brian Gawalt, Laurent El Ghaoui, Luke Barnesmoore, Sophie ClavierarXiv 2014 · arXiv

Introduction

In the contemporary landscape of "big data," researchers and analysts are often overwhelmed by text but lack actionable insight. Traditional methods for summarizing large corpora often fall into two extremes: simple word frequencies (like word clouds), which are often too noisy, or complex generative models (like Latent Dirichlet Allocation), which are computationally expensive and difficult to interpret. This paper proposes Concise Comparative Summaries (CCS), a "Goldilocks" solution that identifies the unique "fingerprint" of a topic relative to a background corpus using a predictive, contrast-based framework.

The CCS Framework

CCS is a lightweight framework designed to generate short, interpretable keyword summaries. Unlike standard frequency-based methods, CCS treats summarization as a classification problem, focusing on what distinguishes a specific topic from the rest of the data.

Methodology

The CCS process involves three primary stages:

  1. Automatic Labeling: Users provide a small set of seed keywords to label documents as "on-topic" (positive) or "off-topic" (negative).
  2. Preprocessing and Weighting: The text is converted into a document-term matrix. The authors utilize $L^2$ rescaling, which proves more robust than tf-idf for short text units by reducing the variance of high-frequency words without losing signal in brief snippets.
  3. Feature Selection via the Lasso: The core of CCS is the use of L1-regularized regression (the Lasso). By treating summarization as a prediction task, the Lasso identifies a sparse set of phrases that best predict the "on-topic" label. This approach accounts for inter-word relationships, reducing redundancy (e.g., picking "United States" as a single concept rather than "United" and "States" separately).

Experimental Validation

The authors validated CCS using two major datasets: the New York Times (NYT) International Section (2009–2011) and headlines from the NYT and Wall Street Journal (WSJ) (2008–2011). A human validation study involving 36 participants was conducted to compare 96 different algorithmic settings.

Key Results

The study found that sparse models like the Lasso and L1-penalized Logistic Regression (L1LR) significantly outperformed traditional co-occurrence and correlation screening in terms of human-rated clarity and relevance. While the Lasso is computationally more intensive than simple counting, it remains far more efficient than complex generative models, making it a practical choice for large-scale applications.

Example of CCS keyword summaries for four countries.

Qualitatively, CCS effectively captures the "framing" of topics. As shown in the example summaries for Iraq, Russia, Germany, and Mexico, the framework identifies specific, distinguishing entities and events—such as "drug cartels" and "sinaloa" for Mexico, or "angela merkel" and "berlin" for Germany—rather than generic international news terms.

Discussion and Conclusion

CCS demonstrates that contrast is essential for effective summarization. By defining a background set, the algorithm ignores general noise and focuses on unique topical markers. While the framework currently relies on initial seed keywords and produces word lists rather than full narratives, it offers a powerful, interpretable tool for navigating massive text databases. Future work aims to extend these techniques to sentence selection to provide even greater narrative context.

Summary updated

Abstract

In this paper we propose a general framework for topic-specific summarization of large text corpora and illustrate how it can be used for the analysis of news databases. Our framework, concise comparative summarization (CCS), is built on sparse classification methods. CCS is a lightweight and flexible tool that offers a compromise between simple word frequency based methods currently in wide use and more heavyweight, model-intensive methods such as latent Dirichlet allocation (LDA). We argue that sparse methods have much to offer for text analysis and hope CCS opens the door for a new branch of research in this important field. For a particular topic of interest (e.g., China or energy), CSS automatically labels documents as being either on- or off-topic (usually via keyword search), and then uses sparse classification methods to predict these labels with the high-dimensional counts of all the other words and phrases in the documents. The resulting small set of phrases found as predictive are then harvested as the summary. To validate our tool, we, using news articles from the New York Times international section, designed and conducted a human survey to compare the different summarizers with human understanding. We demonstrate our approach with two case studies, a media analysis of the framing of "Egypt" in the New York Times throughout the Arab Spring and an informal comparison of the New York Times' and Wall Street Journal's coverage of "energy." Overall, we find that the Lasso with $L^2$ normalization can be effectively and usefully used to summarize large corpora, regardless of document size.
BibTeX
@misc{jia2014concise,
  author = {Jinzhu Jia and Luke Miratrix and Bin Yu and Brian Gawalt and Laurent El Ghaoui and Luke Barnesmoore and Sophie Clavier},
  title = {{Concise comparative summaries (CCS) of large text corpora with a human experiment}},
  year = {2014},
  eprint = {1404.7362},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1404.7362},
}
Figure