A Phrase-Based HMM Approach to Document/Abstract Alignment

Hal Daumé Iii, Daniel MarcuEMNLP 2004 · OpenReview

Abstract

Hal Daumé III and Daniel Marcu introduce a novel Phrase-Based Hidden Markov Model (PBHMM) designed to automatically align documents with their human-written abstracts. By allowing for phrase-to-phrase mappings and non-sequential "jumps," this model significantly outperforms traditional machine translation baselines and previous "cut-and-paste" methods, providing a higher-quality foundation for training statistical summarization systems.

Introduction

Developing automated summarization systems requires large-scale datasets of document-abstract pairs. However, the alignment between a source document and a human-written abstract is often complex and non-linear. Humans do not simply extract sentences; they reorder information, fuse multiple ideas, and paraphrase extensively.

Traditional alignment tools, such as word-based machine translation models or simple extraction models, fail to capture these nuances. Word-based models are too granular, while extraction models incorrectly assume summaries are composed of verbatim sentence snippets—in reality, only about 2.7% of human abstract sentences are pure extracts. To address this, the authors propose a model capable of recognizing phrase-level correspondences across non-contiguous document segments.

The PBHMM Framework

The core of the proposed solution is the Phrase-Based HMM (PBHMM). While a standard HMM emits single words during state transitions, the PBHMM is designed to emit entire sequences or phrases. This architecture allows the model to better reflect human summarization behavior.

State transition diagram for the Phrase-Based HMM.

The model is governed by two primary mechanisms:

  1. The Jump Table: This component models the probability of moving between different positions in the source document. It enables the model to "jump" non-sequentially to find the source of the next summary phrase.
  2. The Rewrite Table: This stores the probabilities of a document phrase being transformed into a specific summary phrase. It also includes a "null" state to account for summary words that have no direct counterpart in the source text (approximately 16% of human abstract words).

To train the PBHMM, the authors utilized an iterative Expectation-Maximization (EM) algorithm. A significant challenge was the model's tendency to align excessively long phrases to maximize probability. This was mitigated through the application of Dirichlet Priors, which nudge the model toward more realistic, human-like alignments by favoring identity or stem-identity matches.

Experimental Evaluation

The model was evaluated using the Ziff-Davis corpus, which contains approximately 2,000 document/abstract pairs related to computer technology. To establish a "gold standard," two human annotators manually aligned 45 pairs. The PBHMM was trained on the 2,000 shortest pairs from the corpus.

Evaluation utilized a "Soft Precision" metric. Because phrase boundaries are often subjective, this metric considers an alignment correct if the underlying words are aligned, even if the specific phrase boundaries differ from the human reference.

Results and Discussion

The PBHMM demonstrated a substantial performance leap over previous techniques. As shown in the results table, it nearly doubled the performance of standard machine translation baselines (GIZA++) and significantly exceeded the "Cut & Paste" method.

Performance comparison of PBHMM against baselines and human performance.

The PBHMM achieved a Soft F-Score of 0.548, compared to 0.363 for the Cut & Paste method and 0.271 for the MT baseline (HMM-flipped). While this represents a major advancement, it still trails human performance, which averages a 0.712 F-Score. Analysis revealed that the model often over-relies on identity matches (e.g., aligning "Microsoft" to "Microsoft" even when context suggests a synonym) and struggles with infrequent synonyms that do not appear often enough to learn strong rewrite probabilities.

Conclusion

The Phrase-Based HMM approach provides a robust framework for document-abstract alignment by moving beyond word-level constraints. By incorporating jump tables and rewrite probabilities with Dirichlet priors, the model captures the non-linear nature of human summarization. Future work aims to integrate coreference resolution and external lexical resources to improve the handling of synonyms and contextual references.

Summary updated

BibTeX
@inproceedings{iii2004phrase,
  author = {Hal Daumé III and Daniel Marcu},
  title = {{A Phrase-Based HMM Approach to Document/Abstract Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2004},
  url = {https://openreview.net/forum?id=BJWITbGdbS},
}
Figure