TL;DR
This paper argues that the standard test set for implicit discourse relation classification is too small to reliably evaluate new features. By performing cross-validation on the entire dataset, the authors demonstrate that features that appear beneficial on the standard test set can show inconsistent or even negative results, highlighting the risk of drawing premature conclusions about feature effectiveness.
What Problem Does This Paper Address?
Discourse relation classification, the task of identifying how sentences or clauses relate to each other (e.g., cause, contrast, elaboration), is crucial for understanding text coherence. A significant challenge in this field is classifying implicit relations, where the connection isn't explicitly marked by a connective word like "because" or "but." Researchers often use the Penn Discourse Treebank (PDTB) for this task, typically training on sections 2-21 and testing on section 23. However, section 23 contains only about 761 implicit relations. This small test set size raises concerns: a minor improvement in accuracy (e.g., 1%) might only correspond to correctly classifying 7-8 additional instances. The authors contend that this limited evaluation set makes it difficult to determine if a newly proposed feature truly offers a general improvement or if the observed gains are merely due to chance or specific properties of that particular test split. This can lead to overestimating the value of certain features and drawing incorrect conclusions about model advancements.
Methods
The core of this paper is a methodological critique and proposal. The authors use a neural network model that incorporates both traditional linguistic features (like Inquirer Tags, Brown Clusters, verb features, Levin classes, and Modality) and learned word embeddings from a Long Short-Term Memory (LSTM) network. This model is designed to be competitive with state-of-the-art approaches.
The key methodological contribution lies in their experimental setup:
- Standard Split: They first replicate the common practice of training on sections 2-21 and testing on section 23 of the PDTB.
- Cross-Validation: They then perform 10-fold cross-validation on the entire corpus (sections 0-24). This approach ensures that all data is used for both training and testing across different folds, providing a more robust estimate of performance. Crucially, they aim for balanced class distributions within each fold to mitigate issues arising from the uneven distribution of relation types.
By comparing the results from these two evaluation strategies, they aim to demonstrate the potential discrepancies in conclusions drawn about feature effectiveness.
Datasets
The primary dataset used is the Penn Discourse Treebank (PDTB). The authors follow standard preprocessing, using sections 2-21 for training and section 23 for testing in the "most-used split." For their cross-validation experiments, they utilize sections 0-24 of the PDTB. They note that the PDTB is the largest available manually annotated corpus for discourse relations, derived from the Wall Street Journal.
Key Insights and Ideas
- The "Small Test Set" Problem: The central argument is that the standard test set (section 23 of PDTB) is too small to reliably assess the generalizability of features for implicit discourse relation classification.
- Risk of Spurious Improvements: Features that show a significant accuracy gain on the standard test set might not generalize well. The authors demonstrate this by showing that some features that appear beneficial on the standard split actually lead to a decrease in performance or inconsistent results across cross-validation folds.
- Cross-Validation as a Solution: The paper strongly advocates for adopting cross-validation as a standard practice in the community. This method provides a more stable and reliable estimate of a model's predictive performance, especially when dealing with limited data.
- Class Imbalance Matters: The authors highlight that the distribution of relation types differs significantly between the training and test sets in the standard split. This imbalance can lead to misestimations of feature usefulness, particularly for under-represented relation types. Their cross-validation approach aims to address this by creating balanced folds.
Key Results
The paper presents results in two tables:
- Table 1: Shows the distribution of training and test sets for the "Most-used Split" versus "Cross Validation." It clearly illustrates that the proportions of relation types are very different between the two. For example, "Temporal.Asynchronous" has a 4.25% training and 1.58% test split in the standard method, while in cross-validation, it's 4.15% train and 4.15% test.
- Table 2: Compares the performance of different features using the "Most-used Split" and "Cross Validation."
- On the Most-used Split, several features (like "Inquirer Tags," "BrownCluster," "Levin Class," "Verbs," "Modality," and "All Features") show accuracies in the range of 37-40%.
- However, when evaluated using Cross Validation, the mean accuracy across folds for "All Features" drops to 35.90% (±1.32), and the standard deviation is significant.
- Specifically, the "Inquirer Tags" feature shows a 2.8% performance increase on the Most-used Split but "actually only helped on two out of 10-fold in the cross-validation set, overall leading to a small decrease."
- Similarly, "verb features" showed a substantial improvement on the standard test set but "there is no effect at all across the folds."
These results strongly support the paper's central claim: the standard test set can be misleading.
Limitations and Future Work
Limitations:
- The authors acknowledge that their cross-validation approach, while better, still uses the same dataset. They suggest that for truly robust evaluation, especially when new datasets are not readily available, cross-validation is a crucial step. They also mention that repeated cross-validation (running the process multiple times with different random splits) could offer an even more thorough evaluation.
- They note that while they aim for balanced distributions in their cross-validation folds, other splitting strategies might be argued for to better represent new in-domain data distributions.
Future Work:
- The paper implicitly calls for the community to adopt cross-validation as a standard practice for discourse relation classification.
- The authors suggest that significance testing, combined with cross-validation, would provide more robust estimates of feature effectiveness and the amount of variation when transferring to new (in-domain) datasets.
Summary updated