Improving Domain Independent Question Parsing with Synthetic Treebanks

Halim Boukaram, Nizar Habash, Micheline Ziadee, Majd F. Sakr

Abstract

Syntactic parsers often struggle with questions because they are typically trained on news articles, which rarely contain interrogative sentences. This research demonstrates that for Modern Standard Arabic (MSA), "synthetic" treebanks—automatically generated training data—can significantly boost parsing accuracy at a fraction of the cost of manual annotation, reducing errors by up to 22% across multiple domains.

The Problem: The Interrogative Domain Gap

Most natural language processing tools are trained on "treebanks" (collections of sentences with mapped grammatical structures) derived from the news domain. Because news reports are primarily declarative, these datasets contain very few questions. For example, the Penn Arabic TreeBank (PATB) contains only 428 questions out of 12,000 sentences.

This leads to a significant "domain gap": a parser trained on news struggles when it encounters a chatbot or a political talk show transcript where questions are frequent. Manually labeling new treebanks is prohibitively expensive and slow. This study investigates whether low-cost, synthetic data generation methods can bridge this gap for Modern Standard Arabic, a language characterized by complex morphology and relatively few digital resources.

Methodology: Building Synthetic Treebanks

The researchers compared three different approaches to building a question-heavy training set:

  1. The Manual Treebank (High Cost): 988 questions were manually annotated from political talk shows and a university "roboceptionist" chatbot to serve as a gold standard.
  2. QGen$_{PATB}$ (Synthetic/Rule-Based): Linguistic rules were applied to existing declarative sentences from the PATB to transform statements into various question types while maintaining correct Arabic morphological agreement.
  3. QTemp (Synthetic/Template-Based): Unlexicalized templates for common chatbot queries (e.g., Where is %place%?) were automatically filled with real-world entities like "cafeteria" or "office."
Figure 4: Example of a declarative statement (a) transformed into a 'What' question (d) using the QGen rule-based method.

The QGen method is particularly powerful as it leverages existing high-quality annotations to create new interrogative structures, such as the transformation from a simple subject-verb-object statement into a complex WH-question.

Figure 3: A synthetic 'Yes/No' question generated using the QGen method, illustrating the inclusion of the 'hal' particle.

Key Insights and Results

The researchers trained the Stanford Parser on various combinations of these datasets, leading to several critical findings:

  • Synthetic Data as a "Booster Shot": Adding synthetic questions to the training mix improved performance across the board. It allows a model to learn the underlying structure of a question even if the specific vocabulary differs from the test set.
  • The "All-In" Strategy: The best-performing model combined baseline news data, manual question data, and both types of synthetic data. This suggests that manual and synthetic data provide complementary benefits.
  • Significant Error Reduction: The combined model achieved a 22% average error reduction across all question-heavy test sets compared to the baseline. On the Chatbot test set specifically, the synthetic QTemp data alone provided an improvement nearly identical to expensive manual data, with the final combined model reducing errors by 40%.
  • Domain Specificity: While synthetic data helped significantly for in-domain tasks, highly specific templates (like those in QTemp) had less impact on unrelated domains like political talk shows.

Conclusion

Synthetic data generation offers a highly cost-effective path to improving parser performance in specialized domains. While the QGen method still requires an initial annotated treebank to transform, it significantly multiplies the value of existing data. Future work will focus on expanding these procedures to cover more complex constructions and testing the generalization of this approach to other languages and downstream tasks like automated Question Answering (QA) systems.

Summary updated

BibTeX
@inproceedings{boukaram2018improving,
  author = {Halim Boukaram and Nizar Habash and Micheline Ziadee and Majd F. Sakr},
  title = {{Improving Domain Independent Question Parsing with Synthetic Treebanks}},
  booktitle = {LAW-MWE-CxG @ COLING},
  year = {2018},
}
Figure