Abstract
Standard Recurrent Neural Networks (RNNs) face significant computational challenges when processing long text sequences due to their sequential nature. While "skipping" mechanisms like Leap-LSTM have been proposed to improve efficiency, they often suffer from a "cold start" problem, where the model lacks guidance on which tokens are critical for classification. This paper introduces Knowledge-aware Leap-LSTM (KALL), which integrates prior human knowledge—such as sentiment lexicons or domain-specific keywords—to supervise the skipping process. KALL achieves up to 2.6x speedups over standard LSTMs and maintains high accuracy even when skipping 90% of the text, outperforming heavy-duty transformers like XLNet in CPU-constrained environments.
Introduction
The primary bottleneck in long text classification is the requirement to process every token in a sequence. While Transformers offer high accuracy, their computational complexity can be prohibitive on standard CPUs. Selective RNNs, such as Leap-LSTM, attempt to "leap" over irrelevant tokens to save time. However, without explicit supervision, these models may inadvertently skip informative words, leading to performance degradation, especially in low-resource scenarios. KALL addresses this by incorporating external knowledge to guide the skipping decision-making process.
Methodology
KALL enhances the Leap-LSTM architecture by introducing mechanisms to utilize keyword information as a form of partial supervision. Two main strategies are proposed for integrating this knowledge:
- Factored KALL (Soft Constraint): Keyword information is represented as a multi-hot vector and concatenated directly with word embeddings, providing a hint to the skipping mechanism about a word's potential value.
- Gated KALL (Hard Constraint): This version utilizes a Keyword Gate and a Decision Gate to mathematically force the model to retain words identified as keywords.
To maintain end-to-end trainability despite discrete "skip or keep" decisions, KALL employs the Gumbel-softmax trick. Furthermore, a Knowledge-Oriented Cost Function ($\mathcal{L}_k$) is introduced to penalize the model if it attempts to skip tokens flagged as important by the human-provided knowledge base.
Experimental Setup and Results
The models were evaluated on four public datasets: AGNews, DBpedia, and reduced versions of Yelp Review Full and Yelp Review Polarity. Knowledge sources included the Opinion Lexicon (for sentiment tasks) and keywords automatically extracted via TF-IDF and TextRank.
As shown in the results, Gated KALL significantly outperforms the standard Leap-LSTM at high skipping rates. For instance, on the Yelp-Polarity Small dataset with a 0.9 skipping rate (reading only 10% of the text), Gated KALL achieved a 92.71 F1 score, compared to 89.06 for the standard Leap-LSTM. In terms of efficiency, KALL demonstrated speedups of 1.1x to 2.6x over standard LSTMs and was over 20x faster than XLNet in CPU-only environments.
Conclusion
KALL effectively bridges the gap between the speed of skipping RNNs and the accuracy of knowledge-guided models. By leveraging existing lexicons and keyword extraction algorithms, it provides a robust solution for long text classification in resource-constrained industrial settings. Future research may extend this framework to other NLP tasks such as document summarization or question answering.
Summary updated