Abstract: While sequence-to-sequence (seq2seq) models achieve state-of-the-art performance in many natural language processing tasks, they can be too slow for real-time applications. One performance bottleneck is predicting the most likely next token over a large vocabulary; methods to circumvent this bottleneck are a current research topic. We focus specifically on using seq2seq models for semantic parsing, where we observe that grammars often exist which specify valid formal representations of utterance semantics. By developing a generic approach for restricting the predictions of a seq2seq model to grammatically permissible continuations, we arrive at a widely applicable technique for speeding up semantic parsing. The technique leads to a 74% speed-up on an in-house dataset with a large vocabulary, compared to the same neural model without grammatical restrictions.
Introduction
Semantic parsing involves translating natural language queries—such as "What are the top five Asian tech companies?"—into executable logical forms (LFs) like SQL or domain-specific query languages. While neural sequence-to-sequence (seq2seq) models are the current standard for this task, their high latency often makes them unsuitable for real-time applications.
The primary performance bottleneck is the softmax operation at the output layer. In complex domains, the vocabulary of possible logical form tokens can exceed 50,000 due to numerous database fields, entities, and enumerated values. Calculating probabilities across this entire vocabulary at every decoding step results in significant latency (250–300 milliseconds per query), hindering the responsiveness of real-time question-answering pipelines.
Methodology
Because logical forms follow strict grammatical rules, this paper proposes exploiting those rules to restrict model predictions. The approach consists of several key components:
- FSA Approximation: The authors approximate the context-free grammar (CFG) of the logical form using a Finite State Automaton (FSA). While an FSA is a simpler formalism, it can represent a superset of the valid grammar, which is sufficient to filter out the vast majority of invalid tokens at any given step.
- Masked Decoding: At each prediction step, the system queries the current FSA state for a list of permissible next tokens (
nextTokens). The neural network's final linear layer is then restricted to calculate scores only for this subset. - The Caching Trick: Dynamically slicing weight matrices in frameworks like PyTorch can introduce significant overhead. To avoid this, the authors precompute and cache reduced matrices for the DFA states, trading memory for a substantial boost in execution speed.
Experimental Setup
The approach was evaluated on an in-house Equity Search (EQS) dataset from Bloomberg, which involves translating financial screening queries into logical forms. The dataset contains 1,981 training pairs and 331 test pairs, with a large vocabulary of 56,209 tokens. This vocabulary is heavily dominated by specific financial metrics, stock indices, and country names.
Results and Discussion
The integration of grammatical constraints yielded significant improvements in both speed and accuracy:
- Efficiency Gains: The baseline neural parser required 0.260 seconds per query. The optimized model with cached grammar integration (NSP-GC) reduced this to 0.067 seconds—a 74% reduction in latency.
- Vocabulary Reduction: On average, the grammar constraints reduced the number of tokens considered at each step from 56,209 to 6,336.
- Accuracy Improvement: By preventing the generation of syntactically invalid logical forms, the model's accuracy on the test set improved from 80.33% to 80.67%.
A critical engineering insight was the "PyTorch Slicing Trap": dynamic matrix slicing without caching actually made the model 17x slower (4.416 seconds) due to framework overhead. This highlights that theoretical reductions in computation do not always translate to real-world speedups without careful implementation.
Conclusion
This work demonstrates that formal grammars are powerful tools for optimizing neural network performance. By using an FSA to approximate the grammar and caching restricted weight matrices, the authors achieved a massive speedup in semantic parsing without sacrificing accuracy. Future work includes moving toward LR grammars for more exact parsing and exploring expressive type-checking systems to further eliminate logical errors.
Summary updated