Abstract
Jesse Dodge, Roy Schwartz, Hao Peng, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
State-of-the-art Natural Language Processing (NLP) models are notoriously over-parameterized, often containing millions or billions of parameters. This leads to massive memory requirements, slow runtimes, and high energy consumption. While researchers often use unstructured sparsity—setting individual weights to zero—to shrink models, standard RNNs like LSTMs and GRUs lack an intuitive internal structure. Pruning them often results in a fragile architecture where removing individual components can lead to a total collapse in performance.
This paper addresses the challenge of structured sparsity: automatically learning which parts of a network's architecture are necessary for a task. The authors introduce a method to learn efficient, compact, and interpretable Recurrent Neural Networks (RNNs) by applying group lasso regularization to rational RNNs.
Methodology
Rational RNNs as WFSAs
The authors leverage Rational RNNs, a family of models where every hidden dimension corresponds to the score of a Weighted Finite-State Automaton (WFSA). This mathematical foundation provides a structured internal representation that is more amenable to pruning than traditional RNN cells.
Structured Pruning via Group Lasso
Instead of penalizing individual weights, the authors apply a "group lasso" penalty to all parameters associated with a specific state in the WFSA.
- State Pruning: If the group lasso penalty pushes a group to zero, that entire state is deleted from the model.
- Dimension Pruning: If all states within a specific WFSA are pruned, the entire hidden dimension is removed. This allows the model to automatically learn the optimal hidden size and n-gram length required for the task.
The Training Pipeline
The architecture learning follows a three-step process:
- Train: Fit the model with the group lasso penalty added to the loss function.
- Prune: Physically remove the states and hidden units whose parameters hit zero.
- Finetune: Re-train the smaller model without the penalty to recover performance.
Experimental Setup
The method was evaluated on binary sentiment classification using the Amazon reviews dataset. Experiments included a mixture of 22 product categories (original_mix) and specific large categories (kitchen, dvd, books). The authors tested the approach using both static GloVe embeddings and frozen contextual BERT embeddings to ensure broad applicability across different input representations.
Results and Discussion
Efficiency vs. Performance
The learned architectures consistently achieved a better tradeoff between accuracy and parameter count than manually tuned baselines. In four out of five experimental setups, the regularized models outperformed unregularized baselines of the same size, sometimes by 1–2% in absolute accuracy. The method successfully produced models that relied on as few as three WFSAs after pruning more than 90% of the weights.
Interpretability and Speed
Unlike Neural Architecture Search (NAS), which can require thousands of GPU hours, this method learns the structure during a standard training run. Furthermore, because rational RNNs are equivalent to WFSAs, the pruned models are human-readable. The authors could visualize specific patterns (e.g., "not worth the time") that the model identified as critical for sentiment analysis. They also noted a "recency preference," where models focused on sentiment-bearing words near the end of documents, aligning with human writing patterns.
Conclusion
This research demonstrates that group lasso regularization on Rational RNNs provides a powerful tool for learning structured sparsity. By automatically pruning unnecessary states and dimensions, the method creates models that are not only significantly smaller and faster but also more interpretable. Future work will look to extend this to more complex tasks like machine translation and explore joint pruning of embedding layers.
Summary updated