Abstract
Neural models for NLP typically use large numbers of parameters to reach state-of-the-art performance, which can lead to excessive memory usage and increased runtime. We present a structure learning method for learning sparse, parameter-efficient NLP models. Our method applies group lasso to rational RNNs (Peng et al., 2018), a family of models that is closely connected to weighted finite-state automata (WFSAs). We take advantage of rational RNNs' natural grouping of the weights, so the group lasso penalty directly removes WFSA states, substantially reducing the number of parameters in the model. Our experiments on a number of sentiment analysis datasets, using both GloVe and BERT embeddings, show that our approach learns neural structures which have fewer parameters without sacrificing performance relative to parameter-rich baselines. Our method also highlights the interpretable properties of rational RNNs. We show that sparsifying such models makes them easier to visualize, and we present models that rely exclusively on as few as three WFSAs after pruning more than 90% of the weights. We publicly release our code.
Introduction
Modern NLP models are notoriously over-parameterized, often containing millions or billions of parameters. While this scale enables state-of-the-art performance, it also results in significant memory overhead, slow inference on edge devices, and a substantial environmental footprint. Structured sparsity techniques, such as group lasso, offer a way to reduce model size by zeroing out entire groups of weights. However, standard recurrent architectures like LSTMs and GRUs lack an intuitive internal structure that lends itself to effective weight grouping.
This paper addresses this challenge by applying structured sparsity to rational RNNs. Unlike standard RNNs, rational RNNs have a hidden state that is explicitly modeled as a collection of weighted finite-state automata (WFSAs). This inherent structure provides a natural way to group parameters for pruning, leading to models that are both compact and highly interpretable.
Methodology: Learning Sparse Structures
The core innovation lies in leveraging the connection between rational RNNs and WFSAs. In a rational RNN, each hidden dimension corresponds to the score of a WFSA reading the input text. These WFSAs learn to recognize "soft" n-gram patterns, such as "such a great [X]" or "not worth [X]".
The authors propose a three-step sparsification process:
- Group Lasso Training: The model is trained with a group lasso penalty. Here, a "group" is defined as all transition parameters associated with entering a specific state within a WFSA.
- State and WFSA Elimination: The penalty pushes the weights of less informative states toward zero. After training, any state with an $L_2$ norm below a threshold ($\epsilon = 0.1$) is removed. If all states in a WFSA are pruned, the entire WFSA is deleted from the architecture.
- Fine-tuning: The resulting sparse architecture is fine-tuned without the regularization penalty to recover any lost performance.
This method effectively performs neural architecture search (NAS) during the training process, determining the optimal number and length of WFSAs without the need for expensive grid searches or thousands of GPU hours.
Experimental Setup
The method was evaluated on the Amazon reviews binary sentiment classification dataset, using several splits: original_mix (a blend of 22 categories) and specific domain splits like kitchen, dvd, and books. The researchers tested the approach using both static GloVe embeddings and contextual BERT embeddings to ensure broad applicability.
Key Results and Insights
The experiments demonstrated that structured sparsity in rational RNNs yields a superior tradeoff between parameter count and accuracy compared to manually sized baselines.
- High Sparsity, High Performance: The method successfully pruned models by over 90%, reducing them to as few as three WFSAs. Despite this extreme reduction, the models maintained strong performance, achieving approximately 88% accuracy on the
original_mixdataset—outperforming unigram baselines by 1-2%. - Enhanced Interpretability: Pruning the model down to a handful of WFSAs makes it possible to visualize exactly what the network has learned. By examining the highest-scoring phrases for specific WFSAs, researchers can identify human-readable sentiment patterns, such as "miserable/returned [X]".
- Embedding Compatibility: While the technique works with both GloVe and BERT, BERT-based models required significantly fewer initial WFSAs (12 vs. 24) to achieve high accuracy, suggesting that contextual embeddings already capture much of the necessary structural information.
Discussion and Limitations
A notable finding involves the interaction between contextual embeddings and interpretability. Because BERT embeddings (specifically the [CLS] token) encode global sentence information, the WFSAs in BERT-based models often rely on these global signals rather than local n-gram patterns. This makes the resulting sparse models less intuitive to visualize compared to those using static embeddings. Future work is needed to develop interpretation methods that can better untangle these complex contextual dependencies.
Conclusion
By applying group lasso regularization to rational RNNs, this work provides a robust framework for learning sparse, interpretable NLP models. The approach eliminates the need for manual architecture tuning while delivering models that are significantly more efficient in terms of memory and computation without sacrificing predictive power.
Summary updated