Knowledge-aware Leap-LSTM: Integrating Prior Knowledge into Leap-LSTM towards Faster Long Text Classification

Jinhua Du, Yan Huang, Karo MoilanenAAAI 2021 · DOI

Abstract

Standard Recurrent Neural Networks (RNNs) face significant computational challenges when processing long text sequences due to their sequential nature. While "skipping" mechanisms like Leap-LSTM have been proposed to improve efficiency, they often suffer from a "cold start" problem, where the model lacks guidance on which tokens are critical for classification. This paper introduces Knowledge-aware Leap-LSTM (KALL), which integrates prior human knowledge—such as sentiment lexicons or domain-specific keywords—to supervise the skipping process. KALL achieves up to 2.6x speedups over standard LSTMs and maintains high accuracy even when skipping 90% of the text, outperforming heavy-duty transformers like XLNet in CPU-constrained environments.

Introduction

The primary bottleneck in long text classification is the requirement to process every token in a sequence. While Transformers offer high accuracy, their computational complexity can be prohibitive on standard CPUs. Selective RNNs, such as Leap-LSTM, attempt to "leap" over irrelevant tokens to save time. However, without explicit supervision, these models may inadvertently skip informative words, leading to performance degradation, especially in low-resource scenarios. KALL addresses this by incorporating external knowledge to guide the skipping decision-making process.

Methodology

KALL enhances the Leap-LSTM architecture by introducing mechanisms to utilize keyword information as a form of partial supervision. Two main strategies are proposed for integrating this knowledge:

  1. Factored KALL (Soft Constraint): Keyword information is represented as a multi-hot vector and concatenated directly with word embeddings, providing a hint to the skipping mechanism about a word's potential value.
  2. Gated KALL (Hard Constraint): This version utilizes a Keyword Gate and a Decision Gate to mathematically force the model to retain words identified as keywords.
The Gated KALL architecture incorporating Keyword and Decision Gates to guide the skipping mechanism.

To maintain end-to-end trainability despite discrete "skip or keep" decisions, KALL employs the Gumbel-softmax trick. Furthermore, a Knowledge-Oriented Cost Function ($\mathcal{L}_k$) is introduced to penalize the model if it attempts to skip tokens flagged as important by the human-provided knowledge base.

Experimental Setup and Results

The models were evaluated on four public datasets: AGNews, DBpedia, and reduced versions of Yelp Review Full and Yelp Review Polarity. Knowledge sources included the Opinion Lexicon (for sentiment tasks) and keywords automatically extracted via TF-IDF and TextRank.

Experimental results comparing KALL variants against standard LSTM and Transformer baselines across various datasets and skipping rates.

As shown in the results, Gated KALL significantly outperforms the standard Leap-LSTM at high skipping rates. For instance, on the Yelp-Polarity Small dataset with a 0.9 skipping rate (reading only 10% of the text), Gated KALL achieved a 92.71 F1 score, compared to 89.06 for the standard Leap-LSTM. In terms of efficiency, KALL demonstrated speedups of 1.1x to 2.6x over standard LSTMs and was over 20x faster than XLNet in CPU-only environments.

Conclusion

KALL effectively bridges the gap between the speed of skipping RNNs and the accuracy of knowledge-guided models. By leveraging existing lexicons and keyword extraction algorithms, it provides a robust solution for long text classification in resource-constrained industrial settings. Future research may extend this framework to other NLP tasks such as document summarization or question answering.

Summary updated

BibTeX
@inproceedings{du2021knowledge,
  author = {Jinhua Du and Yan Huang and Karo Moilanen},
  title = {{Knowledge-aware Leap-LSTM: Integrating Prior Knowledge into Leap-LSTM towards Faster Long Text Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17511},
}
Figure