TL;DR
This paper tackles the challenge of correctly placing adverbial phrases in English sentences, a task crucial for natural language generation. They found that classification-based approaches, particularly a two-stage method, significantly improve adverbial placement accuracy, even outperforming models that use all available features by relying heavily on syntactic information.
What Problem Does This Paper Address?
Surface realization, the process of converting semantic or syntactic representations into natural language, is a fundamental NLP task. While the placement of many sentence elements is relatively fixed in English, adverbial phrases are notoriously flexible. Incorrect placement can lead to sentences that are disfluent, ambiguous, or simply sound "off" to a native speaker. Previous linguistic research has explored the functional and semantic reasons behind adverbial placement, but this paper presents the first computational study to systematically investigate and model this phenomenon using machine learning. The goal is to develop models that can automatically determine the optimal position for adverbials within a sentence.
Methods
The authors compare three approaches to adverbial positioning:
- Baseline: A two-stage approach that first determines an adverbial's position relative to its non-adverbial siblings, and then orders adjacent adverbials pairwise based on frequency in the training data.
- One-stage Classification: A single classifier that predicts the absolute position of all adverbials within a phrase simultaneously. This approach uses a feature vector representing all siblings and labels the output as a permutation of the siblings.
- Two-stage Classification: This approach first classifies an adverbial's position relative to its non-adverbial siblings, and then uses a second stage (identical to the baseline's second stage) to order adjacent adverbials.
All classification approaches utilize a rich set of features, including lexical (word stems, phonemes), syntactic (phrase categories, adverbial types), semantic (hypernyms, number of meanings), and sentence-level features (sentence type, presence of conjunctions). They also explore the impact of using only syntactic features. The models are trained and evaluated using 10-fold cross-validation, with a J4 decision tree classifier being the chosen algorithm.
Datasets
The study uses data extracted from the Wall Street Journal (WSJ) and Switchboard (SWBD) sections of the Penn Treebank III. They specifically extracted adverbial phrases tagged with various adverbial types (e.g., -BNF, -DIR, -EXT, -LOC, -MNR, -PRP, -TMP, -ADV). A portion of this data (10%) was held out for the realization experiment.
Key Insights and Ideas
- Syntactic Features Reign Supreme (Surprisingly): The paper's most striking finding is that classifiers relying solely on syntactic features achieve performance very close to those using all available features (lexical, semantic, and sentence-level). This challenges the intuition that rich semantic and discourse information is strictly necessary for accurate adverbial placement.
- Classification Outperforms Baseline: Both one-stage and two-stage classification-based approaches significantly outperform the baseline method in determining adverbial positions.
- Two-Stage Advantage (for WSJ): The two-stage classification approach shows a slight edge over the one-stage approach, particularly on the WSJ dataset, suggesting a benefit in breaking down the problem into sequential steps.
- Realization Improvement: Integrating the best adverbial positioning models into a surface realizer leads to substantial improvements (at least 10%) in simple string accuracy, demonstrating the practical utility of this research.
Key Results
- Classification Accuracy: The one- and two-stage classification approaches achieve nearly 86% accuracy in determining the absolute position of adverbials on the WSJ dataset.
- Syntactic Feature Performance: Classifiers using only syntactic features performed only about 1-3% worse than those using all features, while producing significantly smaller decision trees.
- Surface Realization Improvement: The two-stage classifier for adverbial positioning improved simple string accuracy by at least 10% in the surface realizer compared to a baseline realizer.
Limitations and Future Work
The authors acknowledge that their feature extraction for semantic and sentence-level information was approximated due to data size constraints. They also note that the performance differences between speech (SWBD) and text (WSJ) data were small.
Future work includes conducting human evaluations to assess the perceived quality of the generated sentences and to explore whether incorporating more features could lead to further performance gains. They also suggest that models of adverbial ordering have broader applications in areas like machine translation, language learning software, and automatic summarization.
Summary updated