Automatic Assessment of Coverage Quality in Intelligence Reports

Samuel Brody, Paul B. KantorACL 2011

Introduction

In the high-stakes domain of intelligence analysis, officers are often inundated with multiple reports from various analysts regarding the same critical topic. Given limited time, the ability to prioritize the highest-quality information is essential. Traditionally, automated quality assessment in Natural Language Processing (NLP) has focused on surface-level features such as grammar, spelling, and vocabulary. However, for an intelligence officer, a report's value lies in its substance—specifically, its content coverage. A report with perfect grammar is of little use if it omits critical information. This research addresses the challenge of measuring "deep" quality: the comprehensiveness and relevance of a document's content.

Methodology

The researchers propose an automated system that treats the assessment of intelligence reports as the inverse of the multi-document summarization task. While summarization aims to extract key themes from a corpus, this system evaluates how many of those essential themes are present in a given report.

The technical approach involves several key steps:

  1. Concept Representation: Concepts are represented as word bigrams (two-word sequences), with common stop words removed.
  2. Prevalence Scoring: In the absence of original source documents, the system analyzes the entire collection of reports for a specific scenario. A "prevalence score" is assigned to every bigram based on its frequency across different analysts' reports, assuming that widely mentioned phrases represent core concepts.
  3. Coverage Scoring: A report is scored by summing the prevalence scores of its unique bigrams. This rewards the breadth of information rather than the repetition of the same concepts.
  4. Ranking: Reports are ranked based on these cumulative coverage scores.

Experimental Setup

The study utilized data from the 2004 ARDA Challenge Workshop, consisting of 56 intelligence reports written by seven analysts across eight different scenarios. To establish a ground truth, each analyst also served as a judge, scoring all reports on a scale of 0–5 across five distinct quality dimensions.

Evaluation criteria used by human judges to assess intelligence report quality.

The primary metric for this study was the average human score for "Coverage." The system's performance was then evaluated using Kendall’s Tau ($\tau_k$), measuring the agreement between the automated rankings and the human-provided gold standard.

Key Results and Insights

The automated concept-based system demonstrated significant efficacy, performing on par with or better than individual human judges.

Agreement levels (Kendall's Tau) across scenarios A-H for word count, human judges, and the concept-based system.

Key findings include:

  • Performance vs. Humans: In four of the eight scenarios, the system's ranking was more accurate than the average human judge. Overall, the system ranked second among all judges (human and automated) in terms of total agreement across scenarios.
  • Beyond Length: While longer reports naturally tend to cover more ground, the concept-based approach consistently outperformed a simple word-count baseline, proving it captures more sophisticated nuances of content quality.
  • Human Variability: The data revealed that human judges often disagree on what constitutes "good coverage," highlighting the cognitive complexity of the task and the value of a consistent automated metric.

Conclusion and Future Work

This research successfully demonstrates that metrics used for generating summaries can be effectively repurposed to evaluate human-written reports. However, several limitations remain. The current sample size is relatively small, and the use of bigrams is a simplified approximation of complex concepts. Future work aims to incorporate more advanced models, such as Latent Dirichlet Allocation (LDA), and expand the system to assess other quality pillars like redundancy avoidance and argumentative structure.

Summary updated

BibTeX
@inproceedings{brody2011automatic,
  author = {Samuel Brody and Paul B. Kantor},
  title = {{Automatic Assessment of Coverage Quality in Intelligence Reports}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2011},
}
Figure