A New Probabilistic Model for Title Generation

Rong Jin, Alexander G. HauptmannCOLING 2002 · OpenReview

Introduction

Automatic title generation is a specialized task in Natural Language Processing (NLP) that sits between simple keyword extraction and full-text summarization. A successful title must not only be relevant to the document's core content but also maintain grammatical fluency. In their paper, Rong Jin and Alexander G. Hauptmann from Carnegie Mellon University propose a refined probabilistic approach that significantly improves upon the "noisy channel" models of the late 1990s. By introducing a hidden "information source" and correcting for statistical biases, their model produces titles that are both more accurate and more natural to human readers.

The Challenges of Title Generation: Noise and Popularity

The authors identify two primary flaws in previous statistical frameworks that hindered the quality of machine-generated titles.

First, there is the "Noise" in Selection. Older models often treated every word in a document as having equal weight in determining the title. However, documents are typically filled with "filler" words and supporting details that do not reflect the main idea. When these common words are allowed to "vote" equally with content-heavy words, the resulting title often misses the point.

Second, the authors highlight a "Popularity Contest" in Word Ordering. Previous models relied on n-gram language models to arrange selected title words into a sequence. Because these language models are trained on large corpora, they naturally assign higher probabilities to common words and phrases. In the context of 1997 news, words like "President" or "Clinton" appeared so frequently that the model would favor them in almost any sequence, regardless of their relevance to the specific document.

Table showing examples of machine-generated titles from baseline models that suffer from the "popularity contest" bias, frequently overusing common terms like "President Clinton".

A Dual-Channel Probabilistic Approach

To address these issues, Jin and Hauptmann proposed a "dual-channel" model that introduces an intermediate hidden state: the Information Source ($S$).

Instead of mapping document words ($DW$) directly to title words ($TW$), the model first "distills" the document into an information source. This is achieved by weighting words using a normalized tf-idf (term frequency-inverse document frequency) metric, ensuring that only the most significant content words participate in the generation process.

Comparison of the structural evolution from the traditional "Old Model" to the proposed "New Model" featuring a hidden Information Source layer.

Furthermore, the authors corrected the word-ordering bias by modifying the probability function used during the sequence generation phase. By dividing the sequence probability $P(T)$ by the probability of the constituent words appearing in any order $P(\{tw \in T\})$, they effectively "subtracted" the influence of word commonness. This forces the model to prioritize the most appropriate arrangement for the specific content rather than the most frequent arrangement in the English language.

Experimental Evaluation and Results

The model was evaluated using a dataset of 50,000 document-title pairs from 1997 broadcast news transcriptions. The researchers compared their new approach against the then-standard Witbrock and Mittal (1999) framework using both quantitative metrics and human assessment.

  • Quantitative Performance: The new model achieved an F1 score of 0.26, a marked improvement over the baseline's 0.21, indicating a higher degree of word overlap with human-written titles.
  • Qualitative Quality: In human evaluations on a 1-5 scale, the new model's titles received an average rating of 3.07, significantly higher than the 2.09 earned by the old model.

The improvement is most evident in specific examples. For a news story regarding a senate vote on tobacco, the old model—biased by word frequency—produced "president clinton budget tax tobacco settlement." In contrast, the new model generated "companies settlement tobacco deal tax laws," which much more accurately reflects the specific subject matter of the report.

Conclusion and Future Directions

While the dual-channel model represents a significant step forward, the authors acknowledge certain limitations. The process of "sequentializing" title words remains computationally expensive, and the current experiments utilized a fixed title length of six words. Future work could explore dynamic title lengths and more complex semantic mappings. Nonetheless, the introduction of the hidden information source and the normalization of ordering probabilities provides a robust foundation for more sophisticated title generation systems.

Summary updated

BibTeX
@inproceedings{jin2002new,
  author = {Rong Jin and Alexander G. Hauptmann},
  title = {{A New Probabilistic Model for Title Generation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2002},
  url = {https://openreview.net/forum?id=cNhIUAKKDk},
}
Figure