Semi-Supervised Learning for Semantic Parsing using Support Vector Machines

Rohit Kate, Raymond MooneyNAACL 2007

TL;DR

This paper introduces SEMISUP-KRISP, a novel semi-supervised learning method for semantic parsing that leverages readily available unannotated sentences to improve parser accuracy. By using transductive Support Vector Machines (SVMs) to retrain classifiers on unlabeled examples extracted from parsing unannotated data, the method significantly boosts performance, especially when labeled training data is scarce.

What Problem Does This Paper Address?

Semantic parsing aims to translate natural language (NL) sentences into formal meaning representations (MRs) that computers can understand and execute. Traditionally, this requires a substantial amount of human-annotated data (NL sentences paired with their correct MRs), which is expensive and time-consuming to create. While unannotated NL sentences are abundant, effectively utilizing them to improve semantic parsers has been a challenge. This paper addresses the gap of leveraging this plentiful unlabeled data to enhance the performance of semantic parsers, particularly in scenarios with limited labeled training data.

Methods

The core of the paper is the SEMISUP-KRISP system, which builds upon the existing supervised semantic parser KRISP. KRISP uses Support Vector Machines (SVMs) with string kernels to learn mappings from NL sentences to MRs, treating each production in the MR grammar as a concept to be classified.

SEMISUP-KRISP enhances KRISP through a semi-supervised approach:

  1. Initial Supervised Training: KRISP is first trained using the available annotated corpus to obtain an initial set of SVM classifiers for each MR grammar production.
  2. Unlabeled Example Extraction: The trained parser is then applied to a set of unannotated sentences. During this process, whenever an SVM classifier is invoked to estimate the probability of a substring representing a semantic concept, that substring is captured as an unlabeled example for that specific classifier. This ensures that the unlabeled examples are representative of the types of inputs the classifier will encounter during real-world use.
  3. Transductive SVM Retraining: The captured unlabeled examples, along with the original positive and negative examples used in the last supervised training iteration, are fed into a transductive SVM algorithm. Transductive SVMs aim to find a labeling for the unlabeled data that maximizes the margin between classes, thereby improving generalization. The paper uses an approximation algorithm for transductive SVMs that doesn't require the proportion of positive and negative examples in the unlabeled data to match the training data.
  4. Final Parser: The retrained classifiers form the final SEMISUP-KRISP parser.

Datasets

The experiments were conducted on the GEOQUERY domain, a standard benchmark for semantic parsing that involves querying a U.S. geography database.

  • Training Data: The paper utilized a corpus of 250 NL queries collected from undergraduate students and annotated with their MRs, along with an additional 630 NL queries collected from a web-based interface and annotated.
  • Unannotated Sentences: The 630 annotated queries were treated as unannotated sentences for the semi-supervised training. An additional 407 queries were collected from the same interface, bringing the total number of unannotated sentences to 1,037.

Key Insights and Ideas

  • Leveraging "In-Distribution" Unlabeled Data: The key insight is that by parsing unannotated sentences with an initial supervised parser, one can extract unlabeled examples that are highly relevant to the task. These examples are "in-distribution" with respect to the test data, making them more valuable than randomly sampled unlabeled data.
  • Transductive SVMs for Semi-Supervised Parsing: The application of transductive SVMs is crucial. Instead of just using unlabeled data to refine class boundaries (like in standard semi-supervised learning), transductive SVMs directly label the unlabeled data in a way that optimizes the overall classifier performance, making them well-suited for this task.
  • Significant Gains with Limited Labeled Data: The method shows particular promise when the amount of labeled training data is small, suggesting it can substantially reduce the human annotation effort required for building accurate semantic parsers.

Key Results

The experimental results demonstrated that SEMISUP-KRISP significantly outperformed the purely supervised KRISP system.

  • Performance Improvement: SEMISUP-KRISP achieved the same performance level (measured by F-measure) as the original KRISP with 25.4% fewer annotated training examples. Specifically, SEMISUP-KRISP reached the performance of KRISP with 126 annotated examples, using only 94 annotated examples itself.
  • Comparison to Hand-Built Parser: The performance of SEMISUP-KRISP was also compared to GEOBASE, a hand-built semantic parser, showing competitive results.

Limitations and Future Work

  • Limitations: The paper acknowledges that the approach relies on the initial supervised parser being reasonably accurate to extract useful unlabeled examples. If the initial parser makes systematic errors, these errors could be propagated. The paper also notes that while they didn't face scaling issues with transductive SVMs in their experiments, scaling to very large datasets might require more advanced optimization techniques.
  • Future Work: The authors suggest that this semi-supervised approach can be extended to other semantic parsing systems and potentially to other NLP tasks that can be framed as classification problems. Further exploration of different transductive SVM variants or other semi-supervised techniques could also be beneficial.

Summary updated

BibTeX
@inproceedings{kate2007semi,
  author = {Rohit Kate and Raymond Mooney},
  title = {{Semi-Supervised Learning for Semantic Parsing using Support Vector Machines}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure