Domain-Independent Shallow Sentence Ordering

Thade NahnsenNAACL 2009

TL;DR

This paper introduces a shallow, feature-based machine learning approach to sentence ordering, demonstrating that simple linguistic and temporal features can achieve competitive performance on event-sequence datasets. While effective for event-driven texts, the model struggles with topic-driven narratives, suggesting that topic modeling is crucial for broader applicability.

What Problem Does This Paper Address?

Ordering sentences correctly within a text is fundamental for natural language understanding and generation tasks, including summarization and question answering. While previous work has explored complex methods, this paper investigates whether a "shallow" approach, relying on readily available linguistic cues and temporal relations, can effectively determine coherent sentence orderings without deep semantic understanding or topic modeling. The authors aim to understand which features are most important and how much performance is gained by incorporating deeper knowledge resources.

Methods

The authors frame sentence ordering as a machine learning problem, specifically a ranking task. They employ a Support Vector Machine (SVM) with a parameter vector w that minimizes pairwise ranking violations in the training data. The core of their contribution lies in the set of features they extract, categorized into three groups:

  • Group Similarity: This category focuses on discourse entities and their occurrences in subsequent sentences. It includes measures of similarity between noun groups, verb groups, and their heads, leveraging WordNet relations (synonym, hypernym, hyponym, antonym) to capture semantic relatedness.
  • Temporal Ordering: This set utilizes the "happens-before" relation from VerbOcean, a resource for mining fine-grained semantic verb relations, to capture temporal precedence between events.
  • Longer Range Relations: This category explores how relations beyond immediate sentence neighbors contribute to coherence. Two approaches are used: counting entity occurrences across a fixed window of sentences and calculating similarity scores between a sentence and others within a range.

Datasets

The paper uses three datasets for evaluation:

  • Earthquakes and Accidents: These datasets, consisting of human-generated texts describing events, were previously used by Barzilay and Lapata (2008). They contain 100 documents each in the training and test sets, with 20 random permutations per text.
  • DUC 2005: This dataset comprises 300 human summaries from the DUC 2005 competition, based on 50 document sets. It results in 6,000 pairwise rankings and is used to evaluate topic-independent properties.

Key Insights and Ideas

  • Shallow Features Can Be Powerful: Surprisingly, simple features related to noun and verb group heads, along with temporal ordering, can achieve good performance, especially on event-driven datasets.
  • Granularity Matters: Considering only the heads of noun and verb groups, rather than the entire groups or sentences, yields the best performance for syntactic units.
  • WordNet's Role: Synonym relations from WordNet improve performance, suggesting that semantic similarity between terms in adjacent sentences is a strong indicator of coherence. However, hypernym and hyponym relations tend to over-generalize.
  • Temporal Cues are Informative: The "happens-before" relation from VerbOcean shows potential for domain-independent sentence ordering, particularly for sequences of events.
  • Topic Dominance: The model struggles significantly on topic-driven datasets (like earthquakes and DUC summaries), indicating that topic modeling is a critical missing component for broader applicability.

Key Results

  • The best performing shallow model, combining temporal and WordNet-based longer-range relations (Chunk+Temp-WN+LongRange+), achieved an accuracy of 87.3% on the accident dataset and 81.0% on the DUC 2005 dataset.
  • On the earthquake dataset, the best shallow model achieved only 14.21% accuracy, significantly underperforming compared to models that incorporate content or coreference information.
  • The temporal ordering features alone achieved 60.41% accuracy on the accident dataset, demonstrating their utility.
  • Cross-training experiments showed that model performance is largely independent of the training topic, but the earthquake dataset remains a challenge.

Limitations and Future Work

The authors acknowledge that their shallow model performs poorly on datasets where sentence ordering is primarily driven by topics, such as the earthquake dataset. They suggest that future work needs to focus on incorporating topic modeling to address this limitation. The paper also notes that while WordNet relations can be helpful, hypernyms and hyponyms may over-generalize. The authors' future work will concentrate on improving performance on topic-driven texts, which is crucial for applications like automatic summarization of newswire articles where topics are unknown during training.

Summary updated

BibTeX
@inproceedings{nahnsen2009domain,
  author = {Thade Nahnsen},
  title = {{Domain-Independent Shallow Sentence Ordering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}

Loading related content...

Figure