Domain-Aware Dependency Parsing for Questions

Yunyao Li, Aparna Garimella, Laura Chiticariu, Yunyao Li 0001ACL 2021 · DOI

TL;DR

This paper tackles the challenge of making AI understand questions in specialized domains, where general-purpose parsers falter. They introduce a novel framework that automatically generates domain-specific question datasets by leveraging existing domain knowledge and a few seed questions. This approach significantly boosts the accuracy of dependency parsers, making AI more adept at handling your niche queries.

What Problem Does This Paper Address?

Building intelligent conversational agents, like Google Assistant or IBM Watson, hinges on accurately understanding user questions. However, natural language processing (NLP) models, particularly dependency parsers, struggle with questions from specific domains (e.g., finance, weather). This is because standard training datasets, like the Penn TreeBank, contain very few questions, and even fewer that are representative of real-world, domain-specific inquiries. Retraining parsers with manually annotated domain-specific questions is prohibitively expensive due to the need for linguistic expertise. This paper aims to bridge this gap by creating a cost-effective method for generating the necessary training data.

Methods

The core of the paper's contribution lies in its automatic labeled domain question generation framework. This framework operates in two main stages:

  1. Template Abstraction: Given a few "seed" domain questions with their syntactic parse trees, the system abstracts them into general question templates. These templates retain the question's structure but replace specific entities with placeholders (e.g., <qsubj>, <qmod>). This is achieved by identifying subject entities (qsubj) and modifier entities (qmod) based on their dependency relations in the parse tree.
  2. Template Filling: The abstracted templates are then populated with new entities derived from domain knowledge. This domain knowledge consists of:
  3. A domain schema: This defines the classes, properties, and relationships within a specific domain (e.g., in finance, classes like "Person" and "Company" with properties like "name" and "title").
  4. A knowledge base: This contains actual data instances that populate the schema (e.g., "Neil Smit" as an instance of "Person").

    The system uses heuristics to fill the placeholders in the templates with relevant information from the schema and knowledge base. Crucially, they also develop methods to automatically construct plausible parse trees for these newly generated questions, ensuring they are suitable for parser training.

    To further enhance the naturalness of generated questions, they introduce schema annotations. These annotations help correct common issues like incorrect prepositions, incomplete property usage, ambiguous semantics, and inappropriate question words by providing additional domain-specific rules and context.

Datasets

  • Existing Datasets:
    • UD Treebank (Universal Dependencies): A general-purpose treebank used for training parsers.
  • QuestionBank: A corpus of 4,000 general questions, also used for training.
  • Newly Introduced Datasets:
    • DQF (Domain-specific questions Finance): 250 questions from the finance domain, manually annotated with dependency parse information.
  • DQw (Domain-specific questions Weather): 250 questions from the weather domain, manually annotated with dependency parse information.
  • NDQ (New Domain Questions): Automatically generated questions using their framework, derived from the DQF and DQw seed sets. These datasets are released to the community.

Key Insights and Ideas

  • Domain Knowledge is Key: The paper strongly emphasizes that leveraging structured domain knowledge (schemas and knowledge bases) is crucial for generating accurate and relevant domain-specific questions.
  • Abstraction and Filling is Powerful: The two-stage process of abstracting questions into templates and then filling them with domain-specific data is an effective way to scale up question generation.
  • Schema Annotations Improve Naturalness: Simply generating questions isn't enough; they need to be natural. The schema annotations are a clever way to inject domain expertise to fix common linguistic errors in generated questions without requiring full manual annotation of every question.
  • Robustness to Seed Size: The framework demonstrates impressive robustness, showing significant improvements even with a very small number of seed questions (as few as 10). This makes it practical for domains with limited initial data.
  • Cost-Effectiveness: The proposed method significantly reduces the effort and cost associated with creating training data for domain-specific parsers, as it automates a large portion of the annotation process.

Key Results

The paper reports significant improvements in dependency parsing accuracy:

  • LAS (Labeled Attachment Score) gains of 4.9% - 9% over parsers trained on general datasets (UD+GQ) when using their auto-generated labeled domain-specific questions (UD+GQ+NDQ).
  • UAS (Unlabeled Attachment Score) gains of 4.8% - 8.8% over general training.
  • The generated questions (NDQ) achieve comparable or better performance than manually annotated domain-specific questions (DQ) when added to general training data, despite being much smaller in size.
  • The framework is robust to seed size, showing improvements even with just 10 seed questions.
  • Schema annotations, while requiring a small one-time effort, lead to 0.4% - 0.94% gains in LAS, further compensating for the annotation effort.

Limitations and Future Work

  • Unnaturalness of Generated Questions: While the schema annotations improve naturalness, the paper acknowledges that generated questions can still exhibit some unnaturalness.
  • Heuristics vs. Learning: The current heuristics for template filling and parse tree construction could potentially be learned from data if sufficient labeled data were available.
  • Future Work: The authors suggest that with enough labeled data, some of these heuristics could be learned automatically. They believe their work paves the way for developing domain-independent question parsing methods that require very little or no training data.

Summary updated

BibTeX
@inproceedings{li2021domain,
  author = {Yunyao Li and Aparna Garimella and Laura Chiticariu and Yunyao Li},
  title = {{Domain-Aware Dependency Parsing for Questions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.400},
}
Figure