Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

Emma Strubell, Patrick Verga, David Belanger, Andrew McCallum

In Natural Language Processing (NLP), balancing computational efficiency with predictive accuracy remains a central challenge. While Bidirectional Long Short-Term Memory (Bi-LSTM) networks have long served as the standard for sequence labeling tasks like Named Entity Recognition (NER), their inherently sequential nature creates a significant bottleneck on modern hardware. Strubell et al. introduce Iterated Dilated Convolutional Neural Networks (ID-CNNs) as a high-performance alternative that leverages the massive parallelism of GPUs to deliver substantial speedups without sacrificing the global context necessary for accurate entity recognition.

The Limitations of Sequential Modeling

To identify entities such as people, locations, or organizations, models must incorporate broad context. For instance, the word "Apple" requires surrounding context to distinguish between a fruit and a corporation. Bi-LSTMs address this by reading sequences forward and backward, but they cannot compute the representation for a token until all preceding tokens have been processed. This linear dependency prevents full utilization of GPU parallel processing power.

While traditional Convolutional Neural Networks (CNNs) are parallelizable, they typically possess a narrow "receptive field," observing only a small window of neighboring words. Expanding this field to cover entire sentences usually requires deep architectures that are prone to overfitting and high parameter counts.

Architecture and Methods

The proposed ID-CNN architecture overcomes these limitations through three primary innovations:

  1. Dilated Convolutions: Unlike standard convolutions, dilated convolutions "skip" tokens at fixed intervals. By stacking layers with exponentially increasing dilation widths (e.g., 1, 2, 4, 8), the model's receptive field grows exponentially with depth. This allows a token to incorporate information from a wide context (e.g., 31 surrounding tokens with just four layers) while maintaining a low parameter count.
Architecture of a dilated convolution showing exponential growth of the receptive field.
  1. Iteration and Parameter Sharing: To refine predictions without increasing model complexity, the authors group dilated layers into blocks and apply the same block multiple times. Sharing weights across these iterations allows the model to perform "depth through refinement," similar to an RNN but with the parallel benefits of a CNN.
  2. Multi-objective Training: The model is trained to predict entity labels after each iteration. This encourages later blocks to act as stacked predictors, refining initial guesses and resolving dependency violations based on the broader context established in earlier passes.

Experimental Results

The models were evaluated on benchmark NER datasets, including CoNLL 2003 and OntoNotes 5.0. The results demonstrate that ID-CNNs achieve state-of-the-art performance with significantly lower latency.

Speed and Efficiency

The primary advantage of the ID-CNN is its speed. Because it utilizes fixed-depth convolutions, computation time is largely independent of sequence length given sufficient GPU memory. The greedy ID-CNN achieved a 14.1x speedup over the standard Bi-LSTM and outperformed the more complex Bi-LSTM-CRF by an even wider margin.

Speed comparison of ID-CNN variants against Bi-LSTM baselines.

Accuracy and Context

On the CoNLL 2003 dataset, the ID-CNN achieved an F1 score of 90.32, nearly matching the Bi-LSTM's 89.34 and remaining competitive with the Bi-LSTM-CRF (90.43). Furthermore, the model's efficiency allows it to process entire documents rather than isolated sentences. When trained on document-level context, the ID-CNN reached an F1 of 90.65, proving its ability to aggregate long-range information that sentence-level models typically miss.

Conclusion

Iterated Dilated CNNs represent a significant step forward for production-ready NLP. By replacing sequential dependencies with parallelizable dilated convolutions, the architecture offers a 14-20x speedup for NER tasks while maintaining accuracy comparable to the Bi-LSTM-CRF. While the addition of a Conditional Random Field (CRF) layer can further improve label consistency, the standalone ID-CNN provides a compelling balance of speed and precision for large-scale entity recognition.

Summary updated

Abstract

Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs. Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining per-token vector representations serving as input to labeling tasks such as NER (often followed by prediction in a linear-chain CRF). Though expressive and accurate, these models fail to fully exploit GPU parallelism, limiting their computational efficiency. This paper proposes a faster alternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. Unlike LSTMs whose sequential processing on sentences of length N requires O(N) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents. We describe a distinct combination of network structure, parameter sharing and training procedures that enable dramatic 14-20x test-time speedups while retaining accuracy comparable to the Bi-LSTM-CRF. Moreover, ID-CNNs trained to aggregate context from the entire document are even more accurate while maintaining 8x faster test time speeds.
BibTeX
@inproceedings{strubell2017fast,
  author = {Emma Strubell and Patrick Verga and David Belanger and Andrew McCallum},
  title = {{Fast and Accurate Entity Recognition with Iterated Dilated Convolutions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1283},
}
Figure