Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks

Chengyue Jiang, Yinggong Zhao, Shanbo Chu, Libin Shen, Kewei TuEMNLP 2020 · OpenReview

Abstract

Neural networks can achieve impressive performance on many natural language processing applications, but they typically need large labeled data for training and are not easily interpretable. On the other hand, symbolic rules such as regular expressions are interpretable, require no training, and often achieve decent accuracy; but rules cannot benefit from labeled data when available and hence underperform neural networks in rich-resource scenarios. In this paper, we propose a type of recurrent neural networks called FA-RNNs that combine the advantages of neural networks and regular expression rules. An FA-RNN can be converted from regular expressions and deployed in zero-shot and cold-start scenarios. It can also utilize labeled data for training to achieve improved prediction accuracy. After training, an FA-RNN often remains interpretable and can be converted back into regular expressions. We apply FA-RNNs to text classification and observe that FA-RNNs significantly outperform previous neural approaches in both zero-shot and low-resource settings and remain very competitive in rich-resource settings.

Introduction

In Natural Language Processing (NLP), practitioners often face a trade-off between the high accuracy of neural networks and the interpretability of symbolic rules. Neural networks are flexible and powerful but suffer from the "cold-start problem," requiring massive labeled datasets to function effectively. Conversely, symbolic rules like Regular Expressions (REs) work instantly (zero-shot) and are human-readable, but they lack scalability and do not naturally improve as more data becomes available.

To bridge this gap, researchers have developed the Finite-Automaton Recurrent Neural Network (FA-RNN). This novel architecture converts REs into trainable RNNs, allowing models to start with expert knowledge, improve through training on labeled data, and remain interpretable enough to be converted back into human-readable rules after the learning phase.

Methodology

The core innovation of the FA-RNN is its multi-step translation process from symbolic logic to neural weights. The process begins by converting a Regular Expression into a Minimal Deterministic Finite Automaton (m-DFA), which is then treated as a Weighted Finite-State Automaton (WFA) with binary weights.

Architecture comparison between a standard RE system and the proposed FA-RNN system.

The recurrence relation in an RNN is mathematically aligned with WFA state transitions. To manage the massive transition tensor ($Vocabulary \times States \times States$), the authors utilize Canonical Polyadic Decomposition (CPD) to break it into three smaller, trainable matrices. This makes the model computationally feasible and compatible with modern deep learning workflows. Additionally, the model can interpolate between RE-based information and standard pretrained word embeddings (like GloVe) to boost performance.

Experimental Results

The FA-RNN was evaluated on three classic text classification benchmarks: ATIS (airline queries), Question Classification (QC), and SMS (spam detection).

Zero-Shot and Cold-Start Performance

In scenarios with minimal labeled data (0% to 1%), FA-RNNs significantly outperformed standard neural networks. For instance, on the ATIS dataset with only 1% of training data, a standard BiGRU achieved near-random accuracy (~1%), while the FA-RNN achieved over 90%. This "warm start" allows the model to match RE performance at step zero and improve immediately upon seeing data.

Rich-Resource Scenarios

Even with 100% of the training data, FA-RNNs remained highly competitive, matching or slightly exceeding the performance of state-of-the-art black-box models. This demonstrates that the rule-based initialization provides a strong inductive bias without hindering the model's ability to learn from large-scale data.

Interpretability and Rule Reconstruction

A unique feature of the FA-RNN is the "interpretability loop." Because the model's weights correspond to automaton transitions, a threshold can be applied to trained weights to reconstruct an improved Regular Expression from the neural network.

An example of a reconstructed FA showing how the model learned new terms like '737' and 'jet' from data.

The reconstructed REs often surpassed the original human-authored ones. On the QC dataset, reconstructed rules were 9.2% more accurate than the originals. As shown in the figure above, a model initialized to identify "aircraft" learned to associate specific flight numbers like "737" or "787" and terms like "jet" or "turboprop" with the category—patterns that were entirely absent from the initial human-written rules.

Conclusion

FA-RNNs provide a powerful framework for combining domain expertise with data-driven learning. While they currently require initial human-written REs and face some complexity trade-offs with gated variants (FA-GRU), they offer a robust solution for low-resource environments. By turning rigid rules into flexible, trainable networks, FA-RNNs ensure that expert knowledge is not lost but rather enhanced by the power of deep learning.

Summary updated

Abstract

Neural networks can achieve impressive performance on many natural language processing applications, but they typically need large labeled data for training and are not easily interpretable. On the other hand, symbolic rules such as regular expressions are interpretable, require no training, and often achieve decent accuracy; but rules cannot benefit from labeled data when available and hence underperform neural networks in rich-resource scenarios. In this paper, we propose a type of recurrent neural networks called FA-RNNs that combine the advantages of neural networks and regular expression rules. An FA-RNN can be converted from regular expressions and deployed in zero-shot and cold-start scenarios. It can also utilize labeled data for training to achieve improved prediction accuracy. After training, an FA-RNN often remains interpretable and can be converted back into regular expressions. We apply FA-RNNs to text classification and observe that FA-RNNs significantly outperform previous neural approaches in both zero-shot and low-resource settings and remain very competitive in rich-resource settings.
BibTeX
@inproceedings{jiang2020cold,
  author = {Chengyue Jiang and Yinggong Zhao and Shanbo Chu and Libin Shen and Kewei Tu},
  title = {{Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=1-72QJuzIQv},
}
Figure