TL;DR
This paper introduces a novel model that unifies local and global features to better understand and predict discourse coherence in text. By combining entity-based local cues with a hidden topic variable for global structure, their model achieves state-of-the-art performance on sentence ordering and significantly outperforms previous methods on a discrimination task.
What Problem Does This Paper Address?
Understanding how sentences connect to form a coherent discourse is crucial for many Natural Language Processing (NLP) tasks, including text generation, summarization, and evaluating written content. Existing models for discourse coherence tend to fall into two camps: local models that focus on relationships between adjacent sentences (e.g., shared words) and global models that track broader topic shifts. Local models are good at keeping related sentences together but struggle with document boundaries and topic changes. Global models, like those based on Hidden Markov Models (HMMs), excel at ordering but can miss fine-grained local connections. This paper aims to bridge this gap by creating a unified model that leverages both local and global information for a more robust understanding of coherence.
Methods
The authors propose a unified model that combines two main components:
- Relaxed Entity Grid (Local Component): This component is adapted from the "entity grid" model (Barzilay & Lapata, 2005). An entity grid tracks nouns and their syntactic roles (subject, object) across sentences. The authors relax independence assumptions in the original model to improve its generative performance. This allows the model to capture how entities and their roles change from one sentence to the next, providing local coherence cues.
- Topic-Based HMM (Global Component): This component is inspired by the HMM content model (Barzilay & Lee, 2004). It introduces a hidden "topic" variable that evolves over sentences, capturing the global structure and topic shifts within a document. The emission probabilities of this HMM are drawn from the relaxed entity grid model, effectively integrating the local entity information into the global topic structure.
The model is trained using a Bayesian approach (non-parametric HMM) and employs Gibbs sampling for inference. They also use backoff and interpolation techniques to handle data sparsity, particularly for lexical information.
Datasets
The primary dataset used for training and evaluation is the AIRPLANE corpus. This corpus consists of documents describing airplane crashes, which are noted for being relatively short (average 11.5 sentences) and formulaic, making them suitable for coherence research. The authors use a standard division of this corpus into 100 training and 100 test documents, with 10-fold cross-validation performed on the training data.
Key Insights and Ideas
- The Power of Unification: The core insight is that combining local entity-level features with global topic-level structure leads to significantly better discourse coherence modeling than either approach alone.
- Relaxing Independence for Generative Models: The authors demonstrate that by relaxing independence assumptions in the entity grid model, they can make it more effective for generative tasks, which is crucial for integrating it into a larger generative framework like an HMM.
- Entity Competition Matters: The relaxed entity grid model addresses a key weakness of the generative entity grid: the assumption that entities appear independently. In reality, nouns compete for syntactic roles, and the relaxed model accounts for this, leading to more realistic probability distributions.
- Topic as a Bridge: The hidden topic variable in the HMM acts as a crucial bridge, allowing the model to capture long-range dependencies and global document structure, while the entity grid provides the fine-grained local context for each topic state.
Key Results
The paper presents results on two main tasks:
- Sentence Ordering: The unified topic-based model achieves state-of-the-art performance with a Kendall's tau (τ) score of .54 on the AIRPLANE test data. This is comparable to the mixture model of Soricut and Marcu (2006) and significantly better than the individual components (Naive Entity Grid: .17, Relaxed Entity Grid: .02, Topic-based (naive): .39).
- Discrimination Task: On the discrimination task, which measures how often a model assigns a higher probability to a correct ordering than to random permutations, the unified model achieves a high score of 96%. This significantly outperforms the global-only model (Barzilay & Lee, 2004) at 74% and also shows improvement over the relaxed entity grid alone (87%).
Limitations and Future Work
- Scalability to Longer Documents: The authors acknowledge that their model, inference algorithms, and evaluation metrics might be difficult to scale up to longer, less formulaic documents.
- Natural Writing Styles: Domains with more natural and varied writing styles present a challenge for lexical prediction.
- Evaluation Metrics for Multi-Paragraph Documents: The τ metric may not be ideal for multi-paragraph documents, as it doesn't distinguish well between transposed paragraphs and transposed sentences. A better approach might involve a coarse-to-fine or hierarchical system.
- Future Work: The authors suggest exploring more complex syntactic features and developing better hierarchical systems for ordering longer documents. They also note that using their model as a component in a mixture model might further improve results.
Summary updated