TL;DR
This paper introduces a novel semi-supervised learning framework that trains statistical models for natural language understanding (NLU) using only abstract semantic annotations, bypassing the need for expensive, fully annotated data. The approach effectively induces semantic parsing rules and demonstrates superior performance over existing methods on the DARPA communicator dataset.
What Problem Does This Paper Address?
Natural Language Understanding (NLU) aims to map sentences to their underlying semantic meaning. Traditional statistical NLU models, like Conditional Random Fields (CRFs) and Hidden Markov Support Vector Machines (HM-SVMs), typically require large amounts of fully annotated data (e.g., explicit word-to-semantic tag alignments) for training. Acquiring such data is costly and time-consuming. This paper tackles the challenge of training these powerful statistical models without this expensive, token-level annotation, instead leveraging "abstract semantic annotations" which describe the semantic structure without explicit word alignments. This is a significant problem because the direct mapping from sentence to abstract annotation is not explicitly provided, making it a "hidden" learning problem.
Methods
The core of the paper is a novel learning framework that uses an Expectation-Maximization (EM) algorithm adapted for training with abstract semantic annotations. The framework operates in several key stages:
- Preprocessing: Abstract semantic annotations are expanded into flattened semantic tag sequences. A "DUMMY" tag is introduced to handle irrelevant words.
- Expectation Step (E-step): This step involves generating candidate semantic tag sequences for each sentence. Crucially, this generation is constrained by the abstract semantic annotations. Two specific constraints are enforced:
- State transitions in the semantic tag sequence are only allowed if they are present in the abstract annotation.
-
Lexical items must be correctly bound to their corresponding semantic tags as defined in the annotation. These constraints are incorporated into the Viterbi algorithm for CRFs and modified discriminant functions for HM-SVMs.
-
Filtering: The generated semantic tag sequences are filtered based on a score that measures their agreement with the abstract semantic annotation. This score is akin to an F-measure, combining precision and recall of semantic tags. Sentences with low-scoring sequences are discarded.
- Maximization Step (M-step): The filtered semantic tag sequences and their corresponding sentences are used to re-estimate the model parameters using standard training algorithms (maximum likelihood for CRFs, and optimization for HM-SVMs).
This iterative process allows the models to learn robust semantic parsing capabilities from less detailed annotations.
Datasets
The primary dataset used for training and evaluation is the DARPA communicator data. This dataset consists of spoken language utterances and their corresponding abstract semantic annotations. The authors note that these annotations describe embedded structural context but do not provide explicit word/semantic tag alignments, making them more complex than datasets like CLASSIFIED (used in prior work) which lack hierarchical relations. For their experiments, 46 days of data were randomly selected for testing, with the remaining data used for training.
Key Insights and Ideas
- Abstract Annotations are Sufficient: The paper demonstrates that valuable semantic parsing models can be trained effectively using only abstract semantic annotations, significantly reducing the annotation burden.
- EM with Constraints is Powerful: Adapting the EM algorithm to incorporate specific constraints derived from abstract annotations is key to learning accurate semantic representations.
- Filtering is Crucial: The filtering step, which prunes low-quality generated semantic sequences, significantly boosts performance, highlighting its importance in the learning pipeline.
- HM-SVMs Excel with Abstract Data: HM-SVMs, with their ability to learn nonlinear discriminant functions, show a slight advantage over CRFs when trained with this framework, suggesting their suitability for complex semantic structures.
Key Results
- The proposed framework, when applied to CRFs and HM-SVMs, outperforms the baseline Hidden Vector State (HVS) model trained on similar abstract annotations.
- The framework achieves superior performance compared to two other baseline approaches: a hybrid framework combining HVS and HM-SVMs, and discriminative training of HVS.
- Specifically, the proposed approach achieved a relative error reduction of approximately 25% and 15% in F-measure when compared to the hybrid and discriminative HVS training methods, respectively.
- The filtering step was found to be crucial, boosting performance by nearly 4% for CRFs and 3% for HM-SVMs.
- Performance generally peaked around iteration 8 of the training process, with F-measures reaching over 92% for CRFs and 93% for HM-SVMs.
Limitations and Future Work
- Feature Engineering: While CRFs and HM-SVMs can handle arbitrary input features, the paper notes that the HVS model has limitations in incorporating them. The proposed framework aims to bridge this gap.
- Future Work: The authors suggest exploring other scoring functions for the filtering step to better describe the precision of parsing results. They also plan to apply the framework to other domains like information extraction and opinion mining.
Summary updated