Abstract
Semantic parsing typically requires large, fully-annotated corpora, which are expensive and time-consuming to produce. This paper introduces a hybrid framework that leverages a generative model to bootstrap a discriminative model, drastically reducing human annotation effort. By selecting representative sentences through clustering and using "light" abstract annotations, the system auto-generates training data for a high-performance Hidden Markov Support Vector Machine (HM-SVM). Evaluation on the DARPA Communicator dataset demonstrates that this hybrid approach achieves an 88.5% F-measure, outperforming traditional models while requiring significantly less manual labor.
Introduction
Semantic parsing—the task of translating natural language into formal meaning representations—is a cornerstone of modern language processing. Traditionally, researchers choose between generative models, which can handle "light" abstract annotations but often lack peak accuracy, and discriminative models, which offer high performance but require exhaustive word-level mappings.
The primary challenge addressed here is the annotation bottleneck: how to achieve the superior performance of discriminative models without the prohibitive costs of full, word-level human annotation. The authors propose a solution that bridges these two paradigms, using the strengths of generative modeling to automate the data requirements of discriminative learning.
Methodology
The proposed framework follows a multi-stage pipeline designed to minimize human intervention while maximizing data utility.
- Representative Sentence Selection: To avoid the inefficiencies of random sampling, a modified K-means clustering algorithm based on sequence alignment similarity identifies structural centroids in an un-annotated corpus.
- Light Human Annotation: Human annotators provide "abstract annotations" (semantic concepts without word-level alignment) only for these selected representative sentences.
- Generative Bootstrapping: A generative Hidden Vector State (HVS) model is trained on this small, lightly annotated set.
- Auto-Generation: The HVS model parses the remaining un-annotated corpus. High-confidence results are filtered to create a large, automatically "fully" annotated dataset.
- Discriminative Training: A powerful HM-SVM is trained on this auto-generated corpus to produce the final semantic parser.
Experimental Setup
The framework was evaluated using the DARPA Communicator Data, a collection of travel-related utterances (e.g., flight bookings). The training set consisted of 12,702 utterances, with a separate test set of 1,178 utterances. Performance was measured using the F-measure on extracted slot/value pairs.
Results and Discussion
The hybrid framework demonstrated significant improvements in both efficiency and accuracy.
Efficiency Gains
One of the most striking findings was the reduction in manual labor. Abstract annotation proved to be approximately six times faster than traditional word-level annotation. As shown in Figure 5, the time required to annotate sentences grows much more slowly when using the abstract method compared to the word-level method.
Specifically, fully annotating 350 sentences at the word level required 17.5 hours, whereas abstract annotation for the same set took only 3 hours.
Performance Accuracy
Despite the reduced human input, the hybrid model achieved an 88.5% F-measure using only 2,763 abstractly annotated sentences. This represents a 12% relative error reduction over the baseline HVS model (87.0%) trained on the same data. Remarkably, this hybrid model also outperformed a baseline HVS model trained on the entire dataset of 12,702 sentences (87.97%), proving that smart sampling and bootstrapping can be more effective than sheer volume.
Conclusion
This research proves that generative models can serve as effective "teachers" for more complex discriminative models. By automating the transition from abstract concepts to word-level alignments, the hybrid framework offers a scalable path for developing semantic parsers in new domains.
Future work may address the system's sensitivity to outliers in the clustering phase, as rare structural patterns can occasionally degrade performance when selected as centroids. Refining the confidence thresholds for auto-generation also remains an area for further optimization.
Summary updated