Introduction
Building effective Virtual Assistants (VAs) like Alexa or Siri requires robust Natural Language Understanding (NLU). This usually involves two primary sub-tasks: Intent Detection (identifying the user's goal, e.g., "SearchFlight") and Slot Filling (extracting specific entities, e.g., "Toronto" as a departure city).
While Transformer models are powerful, they often struggle with complex sentence structures or ambiguous words. Previous attempts to incorporate "syntactic knowledge"—grammar rules and sentence structure—usually required running a separate, heavy dependency parser during inference. This adds significant latency and computational cost. This paper introduces a "syntactically-informed" Transformer architecture that bakes grammatical knowledge directly into its attention mechanism during training, requiring no external linguistic tools at runtime.
Proposed Architecture: A Syntactically-Informed Transformer
The authors propose a novel Transformer encoder-based architecture that utilizes Multi-Task Learning (MTL). The model is trained on four tasks simultaneously to internalize linguistic structure:
- Intent Detection (ID): Utterance-level classification.
- Slot Filling (SF): Token-level classification.
- Part-of-Speech (POS) Tagging: Identifying word categories (noun, verb, etc.).
- Syntactic Dependency Prediction: Learning the grammatical relationships between words.
The Syntactically-Informed Layer
In a standard Transformer, attention heads are free to attend to any part of the sentence. In this model, the authors "force" one specific attention head to learn the dependency structure of the sentence. Unlike previous work that only predicted a word's direct "parent" in a grammar tree, this model is trained to predict the entire lineage (parents, grandparents, etc.) through Ancestor Prediction.
During training, the model compares its internal attention weights against a "prior" distribution derived from a pre-trained dependency parser. It uses KL Divergence Loss to minimize the difference, effectively "teaching" the attention head to focus on syntactically relevant words. Crucially, because this is handled via the attention mechanism, the syntactic supervision is only needed during training; at inference time, the model simply uses its learned attention weights.
Key Insights and Ideas
- Syntax as a Training Scaffold: By using a parser as a "teacher" during training, the model internalizes structural rules, removing the need for a parser at runtime.
- The Power of Ancestry: Predicting the full syntactic ancestry helps the model capture long-distance relationships in a sentence, which is vital for slot filling in complex queries.
- Solving the Preposition Trap: Models without syntactic knowledge often fail on prepositions. For example, in "Kansas City to Atlanta," the word "to" is the key to distinguishing the departure city from the arrival city. Syntactic supervision helps the model link these nouns correctly.
Key Results
The "SyntacticTF" model achieved impressive results, outperforming previous state-of-the-art models that did not use massive pre-training:
- SNIPS: Achieved an absolute F1 score improvement of 1.59% for slot filling and 0.85% accuracy improvement for intent detection.
- ATIS: Improved slot filling by 0.1% F1 and intent detection by 0.34%.
- Efficiency: The model is significantly smaller than heavyweights like BERT. It achieved results competitive with JointBERT while being roughly 1/7th the size (16 million vs. 110 million parameters).
Limitations and Future Work
The authors noted that on the SNIPS dataset, adding dependency prediction alone (without POS tagging) slightly decreased slot-filling performance. This may be due to SNIPS' shorter sentences and complex named entities (like song titles) which can introduce noise into standard dependency parsers. Future work includes testing the architecture on larger, more linguistically complex datasets and exploring how to incorporate this syntactic supervision into pre-trained models like BERT or RoBERTa.
Summary updated