Abstract
Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs. Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining per-token vector representations serving as input to labeling tasks such as NER (often followed by prediction in a linear-chain CRF). Though expressive and accurate, these models fail to fully exploit GPU parallelism, limiting their computational efficiency. This paper proposes a faster alternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. Unlike LSTMs whose sequential processing on sentences of length N requires O(N) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents. We describe a distinct combination of network structure, parameter sharing and training procedures that enable dramatic 14-20x test-time speedups while retaining accuracy comparable to the Bi-LSTM-CRF. Moreover, ID-CNNs trained to aggregate context from the entire document are even more accurate while maintaining 8x faster test time speeds.
Introduction
Bi-directional LSTMs (Bi-LSTMs) have become the standard for sequence labeling tasks such as Named Entity Recognition (NER). While expressive, their sequential nature creates a computational bottleneck; processing a sequence of length $N$ requires $O(N)$ time, failing to fully exploit the parallel processing capabilities of modern GPUs. As NLP is increasingly applied to massive datasets, the need for faster, parallelizable alternatives has become paramount.
Iterated Dilated Convolutional Neural Networks
To address these efficiency concerns, this paper introduces Iterated Dilated Convolutional Neural Networks (ID-CNNs). Unlike LSTMs, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents.
The architecture relies on three key innovations:
- Dilated Convolutions: By using dilated convolutions that skip tokens in a sliding window, the model's receptive field grows exponentially with depth. This allows a few layers to capture broad context without the parameter overhead of standard CNNs.
- Iterated Blocks: To prevent overfitting while maintaining depth, the model applies a single block of dilated convolutions iteratively, sharing parameters across iterations. This allows the network to build broad context efficiently.
- Intermediate Supervision: A loss function is applied after each block iteration during training. This forces early blocks to make good initial predictions, while subsequent blocks learn to refine them, effectively mimicking the structured reasoning of a CRF within the feature extraction phase.
Experimental Results
The ID-CNN was evaluated on the CoNLL-2003 and OntoNotes 5.0 benchmarks. The results demonstrate that ID-CNNs can match the accuracy of Bi-LSTM-CRF models while providing significant speed advantages.
Speed and Efficiency
On the CoNLL-2003 dataset, a sentence-level greedy ID-CNN achieved an F1 score of 90.32, nearly identical to the 90.43 achieved by a Bi-LSTM-CRF. Crucially, the ID-CNN architecture is significantly faster. As shown in the table below, the ID-CNN achieves a 14.10x speedup over the Bi-LSTM-CRF baseline at test time.
Document-Level Context
The efficiency of ID-CNNs allows them to process entire documents (1,000+ tokens) simultaneously. When trained with document-level context, the ID-CNN's F1 score on CoNLL-2003 improved to 90.65, outperforming the Bi-LSTM-CRF while remaining nearly 8x faster.
Conclusion
ID-CNNs offer a compelling alternative to recurrent architectures for sequence labeling. By combining dilated convolutions with parameter sharing and intermediate supervision, they achieve state-of-the-art accuracy with order-of-magnitude improvements in inference speed. Future work includes integrating character-level representations or hand-crafted lexicons to further enhance performance.
Summary updated