Using Phrasal Patterns to Identify Discourse Relations

Manami Saito, Kazuhide Yamamoto, Satoshi SekineNAACL 2006

TL;DR

This paper tackles the challenge of automatically identifying how sentences relate to each other in Japanese text. By combining traditional lexical analysis with a novel approach using "phrasal patterns" (sequences of phrases), the researchers significantly improved accuracy in classifying these discourse relations, achieving performance comparable to human annotators.

What Problem Does This Paper Address?

Understanding how sentences connect is crucial for many Natural Language Processing (NLP) tasks, like text summarization and question answering. These connections, known as discourse relations, explain the logical flow of text (e.g., cause-effect, contrast, elaboration). While previous work has shown the importance of lexical cues (individual words), identifying these relations, especially in languages like Japanese with flexible word order and rich morphology, remains a challenge. This paper specifically addresses the problem of automatically identifying discourse relations between consecutive sentences in Japanese, aiming to improve upon existing methods by incorporating structural linguistic information.

Methods

The authors propose a system that combines two main approaches:

  1. Lexical Information: This is a standard approach where word pairs from consecutive sentences are analyzed. They extract common nouns, verbal nouns, verbs, and adjectives, considering their part-of-speech and annotating them with positive/negative labels based on the presence of negation. Frequencies of these word pairs within different discourse relations are used to calculate scores.
  2. Phrasal Pattern Information (Novelty): This is the core contribution. Instead of just individual words, they identify recurring sequences of phrases (called "bunsetsu" in Japanese) that span across two sentences. These patterns are constructed by:
  3. Deleting unnecessary phrases: Removing modifiers and redundant clauses to focus on core structural elements.
  4. Restricting phrasal patterns: Applying a regular expression to ensure patterns are meaningful and relevant.
  5. Combining phrases: Building patterns from at least three subphrases, with at least one from each sentence. These phrasal patterns are then used to calculate scores, similar to the lexical approach, but using pattern frequencies instead of word pair frequencies.

The system then combines scores from both lexical and phrasal pattern analysis to make its final prediction.

Datasets

  • Corpus: The system was trained on a large corpus derived from the Web (approximately 20GB of text) and 38 years of newspapers.
  • Extracted Sentence Pairs: Approximately 1.3 million sentence pairs from the Web and 150,000 pairs from newspapers were extracted, specifically focusing on pairs where an unambiguous discourse cue phrase appeared at the beginning of the second sentence.
  • Test Corpus: 300 sentence pairs were set aside for testing, with 50 pairs for each of the six discourse relations (excluding "OTHER").

Key Insights and Ideas

  • Phrasal structure matters: The most significant insight is that the structure of phrases, not just the individual words, carries strong signals for discourse relations. This is particularly relevant for languages like Japanese where word order can be flexible.
  • Beyond individual words: Relying solely on lexical information can miss crucial relational cues. Phrasal patterns capture more abstract structural relationships that are indicative of discourse connections.
  • Human-level performance: The system's accuracy, especially when using phrasal patterns, approaches that of human annotators, suggesting the approach is capturing meaningful linguistic phenomena.
  • Frequency vs. Signal: The authors acknowledge that high-frequency discourse relations (like ELABORATION and CONTRAST) are easier to identify. Their scoring mechanisms (Score2 and Score4) attempt to balance this by adjusting for overall corpus frequency, helping to identify less frequent relations.

Key Results

  • Accuracy Improvement: Incorporating phrasal pattern information improved the system's accuracy by 12 percentage points, from 53% (lexical only) to 65% (lexical + phrasal).
  • Performance Comparison: The weighted accuracy of the system using both methods (65%) is comparable to human performance (average inter-annotator agreement of 69%, and system accuracy of 63% when compared against three human subjects).
  • Relation-Specific Gains: Phrasal patterns showed particularly strong gains for relations like CONTRAST (from 62% to 86% accuracy).
  • Baseline: The baseline accuracy (always selecting the most frequent relation) was 43%, highlighting the significant improvement achieved.

Limitations and Future Work

  • Limitations:
    • The system primarily focuses on consecutive sentences, limiting its ability to capture discourse relations across longer spans of text.
  • The authors acknowledge that the system makes mistakes, and even human annotators struggle when cue phrases are absent.
  • The "OTHER" discourse relation category was ignored due to its low frequency.
  • Future Work:
    • Applying other machine learning methods.
  • Analyzing and refining the discourse relation categorization strategy.
  • Extending the context beyond just two sentences to capture broader discourse structures.

Summary updated

BibTeX
@inproceedings{saito2006using,
  author = {Manami Saito and Kazuhide Yamamoto and Satoshi Sekine},
  title = {{Using Phrasal Patterns to Identify Discourse Relations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure