TL;DR
This paper introduces a method for learning semantic grammars from weakly supervised, ambiguous data, enabling them to be used for both speech recognition and semantic parsing simultaneously. This approach reduces the need for extensive manual annotation, a major bottleneck in spoken language understanding systems, and shows competitive performance compared to traditional methods.
What Problem Does This Paper Address?
Spoken Language Understanding (SLU) systems typically rely on a pipeline: first, an Automatic Speech Recognizer (ASR) transcribes spoken input, and then a semantic parser maps this transcription to a formal meaning representation. A major challenge is that ASR errors can significantly degrade the performance of the downstream semantic parser. Traditionally, semantic grammars, which can bridge acoustics and semantics, are either manually created or learned in a supervised manner, both requiring substantial effort. This paper addresses the problem of acquiring these semantic grammars more efficiently by using "weak supervision" – ambiguous contextual information – to learn grammars that can serve dual roles: as a language model for the ASR and as a semantic parser for understanding.
Methods
The core of the paper is the induction of semantic grammars from weakly supervised data. The authors adapt an existing algorithm for semantic parser induction, originally designed for textual input, to learn grammars from ambiguous textual utterances paired with predicate logic representations. This process involves:
- Initial Lexicon Acquisition: Learning potential mappings between words/phrases in the text and semantic referents by analyzing co-occurrence frequencies.
- Alignment Computation: Creating hypotheses about how parts of the textual utterance map to semantic referents within the given ambiguous context.
- Co-occurrence Frequency Estimation: Estimating associations between lexical units and semantic referents, syntactic patterns and semantic frames, and syntactic slots and semantic argument slots.
The learned grammars are then transformed into a format (JSGF) suitable for speech recognition. Crucially, the paper explores two types of grammars: unweighted and weighted. Weighted grammars incorporate occurrence frequencies from the training data, giving more importance to rules that appeared more often. The paper also compares these grammars against traditional n-gram language models.
Datasets
The experiments were conducted on the RoboCup soccer corpus. This dataset consists of human-commented soccer games, where utterances are paired with predicate logic formulas representing game events. The corpus provides both ambiguous training data and a gold standard for evaluation. The authors used a 4-fold cross-validation scheme, training on three games and testing on the fourth.
Key Insights and Ideas
- Weak Supervision is Viable: Learning semantic grammars from ambiguous context (weak supervision) is feasible and can yield grammars useful for both ASR and semantic parsing. This significantly reduces the annotation burden.
- Joint Models Outperform Cascading: Using the same semantically motivated grammar for both speech recognition and understanding (a joint approach) leads to better parsing performance compared to a cascaded system where a standard n-gram model is used for ASR.
- Weighting Matters: Incorporating occurrence frequencies as weights in the grammar rules improves parsing performance, suggesting that prioritizing more common linguistic patterns is beneficial.
- WER vs. Understanding: A lower Word Error Rate (WER) in speech recognition does not automatically guarantee better semantic parsing performance. The type of errors made by the ASR is critical; errors that confuse semantically important words can be more detrimental than others. Semantic grammars, by explicitly modeling semantic relationships, can be more robust to certain types of ASR errors.
Key Results
- Semantic Parsing Performance: The best performing model was the weighted semantic grammar combined with a trigram back-off model, achieving an F₁ score of 84.46% on speech data. This is a significant improvement over the baseline trigram model alone (78.36% F₁).
- Grammar vs. N-gram: Unweighted semantic grammars (without back-off) achieved 82.37% F₁, outperforming the baseline trigram model.
- Weighted vs. Unweighted Grammars: Weighted semantic grammars (without back-off) achieved 84.46% F₁, outperforming unweighted ones (82.24% F₁).
- Speech Recognition (WER): While semantic grammars alone sometimes increased WER compared to trigrams, their combination with trigram back-off models showed competitive or improved WER in some cases. The best WER was achieved by the trigram baseline (7.1%), but this came at the cost of parsing performance.
Limitations and Future Work
- OOG Utterances: The paper notes that a decrease in parsing precision for weighted semantic grammars (when used with a back-off trigram) was partly due to Out-Of-Grammar (OOG) utterances being matched incorrectly by the ASR. Tuning recognition parameters could potentially mitigate this.
- Parameter Tuning: The experiments were conducted without extensive optimization of recognition parameters, suggesting that further improvements might be possible.
- Semantic Classes: The authors suggest exploring the use of semantically induced classes within class-based language models as a future direction.
- Combining Models: While the paper shows that combining semantic grammars with trigram models can be beneficial, it notes that most ASR systems typically use one LM at a time. Future work could explore more sophisticated ways to combine these models.
- Manual Grammar Comparison: The authors acknowledge that a comparison with a manually created gold standard grammar would be valuable, but such a grammar was not available for the dataset.
Summary updated