NSTM: Real-Time Query-Driven News Overview Composition at Bloomberg

Joshua Bambrick, Minjie Xu, Andy Almonte, Igor Malioutov, Guim Perarnau, Vittorio Selo, Iat Chong ChanACL 2020

TL;DR

Bloomberg's NSTM system tackles the overwhelming volume of daily news by providing concise, query-driven overviews. It leverages advanced semantic clustering and novel summarization techniques to distill millions of articles into digestible "themes," making it easier for users to grasp key events quickly.

What Problem Does This Paper Address?

In today's information-saturated world, consuming the sheer volume of news generated daily is a monumental challenge. For instance, a simple search for "Apple Inc." can yield thousands of articles, many of which are redundant or have sensationalized headlines, making it incredibly difficult for users, such as financial analysts, to quickly identify the most important information and understand the "why" behind market movements. Traditional search systems often return a list of links, forcing users to sift through duplicates and uninformative content. This paper addresses the need for a system that can intelligently filter, cluster, and summarize this deluge of news into a coherent and actionable overview, tailored to a user's specific query.

Methods

NSTM employs a multi-stage approach to achieve its goal of real-time news overview composition:

  • Story Ingestion and Online Clustering: News articles are continuously ingested and initially clustered using an online, incremental clustering algorithm. This pre-processing step reduces the computational load for subsequent query-time processing.
  • Query-Time Hierarchical Agglomerative Clustering (HAC): When a user submits a query, NSTM retrieves relevant articles and their pre-computed online cluster IDs. It then performs a second stage of clustering using HAC on these online clusters to form more refined, query-specific "themes."
  • News Embedding and Similarity: The core of the clustering relies on NVDM (Neural Variational Document Model) embeddings, which capture semantic similarity between articles. Cosine similarity is used to measure the closeness of these embeddings.
  • Summary Generation: For each cluster (theme), NSTM generates a concise summary using a two-pronged approach:
  • OpenIE-based Tuple Extraction: This symbolic method extracts predicate-argument tuples from sentences, which are then pruned and converted into surface forms to create candidate summaries.
  • BERT-based Sentence Compression: This neural approach uses a fine-tuned BERT model to classify tokens within a sentence as "keep" or "delete" to create compressed, high-quality summaries.
  • Candidate Ranking: A sequence-pair model is trained to rank the generated summary candidates, selecting the best one for each theme. This model is trained on a dataset of human-annotated summaries.
  • Key Story Selection: Within each theme, NSTM selects a few representative "key stories" by re-clustering the stories in the theme and picking the story most similar to others in its sub-cluster.
  • Theme Ranking: Finally, themes are ranked based on their size (number of stories) and the entropy of their news sources, aiming to prioritize more important and diverse topics.

Datasets

  • Training Data:
    • A corpus of 1.85 million news articles was used to train the NVDM model.
  • A custom dataset of 10,000 annotations was created for the BERT-based sentence compression task, where annotators selected sentences and deleted words to create summaries.
  • A dataset of approximately 48,000 pairwise samples (candidate summary, article) was created for training the summary candidate ranking model, with human annotations labeling candidates as "Great," "Acceptable," or "Terrible."
  • Evaluation Data:
    • A manually annotated dataset of over 1,000 news article pairs was used to determine the optimal similarity threshold for HAC.
  • The paper also mentions using ROUGE scores for evaluating the end-to-end system's summarization performance, referencing a Single Document Summarization (SDS) test set.

Key Insights and Ideas

  • The Power of Two-Stage Clustering: Using an initial online clustering at ingestion time significantly reduces the computational burden at query time, while a subsequent HAC step refines these clusters into more meaningful themes. This hybrid approach balances efficiency and quality.
  • Leveraging Both Symbolic and Neural Summarization: Combining OpenIE (symbolic) and BERT-based sentence compression (neural) for summary candidate generation creates a more diverse pool of potential summaries. This diversity allows the ranker to select the most appropriate summary, rather than being limited by the strengths of a single method.
  • Focus on "Informational" Summaries: The ranking mechanism, which favors summaries that are representative of the story body rather than just headlines, naturally leads to more informational and less sensational summaries. This is crucial for users who need to make informed decisions.
  • Proxy Metrics for Importance: Ranking themes by cluster size and source entropy provides a practical way to estimate the importance of a news topic, reflecting both its prevalence and the diversity of its reporting.

Key Results

  • NSTM is deployed to hundreds of thousands of users and handles thousands of requests daily with sub-second latency.
  • 99.6% of requests hit the cache, with 99% handled within 215ms. Cache misses result in a median response time of 723ms.
  • The end-to-end system, when using both OpenIE and BERT-based sentence compression, achieved ROUGE-L F1 scores of 0.670 on the Single Document Summarization task, outperforming either method used alone.
  • Qualitative feedback from users indicates that over three-quarters of submissions are positive.

Limitations and Future Work

  • Summary Length Constraint: The paper mentions a length constraint of 50 characters for summaries, which might be too restrictive for some complex topics.
  • Autoregressive Sentence Compression: The authors suggest exploring whether autoregressive approaches in neural sentence compression can be further exploited.
  • Longer Time Periods: Composing themes over longer time periods is identified as an area for future research.
  • Data Scarcity for Abstractive Summarization: The paper notes the difficulty of training end-to-end abstractive summarization models due to a lack of training data.
  • Potential for Over-fragmentation: While HAC helps, the initial online clustering can lead to over-fragmentation, which might still impact the quality of some themes.

Summary updated

BibTeX
@inproceedings{bambrick2020nstm,
  author = {Joshua Bambrick and Minjie Xu and Andy Almonte and Igor Malioutov and Guim Perarnau and Vittorio Selo and Iat Chong Chan},
  title = {{NSTM: Real-Time Query-Driven News Overview Composition at Bloomberg}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure