A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021

Mark Anderson, Carlos Gómez-RodríguezACL 2021

TL;DR

This paper presents a modest Pareto optimization analysis of three modern dependency parsing paradigms (Biaffine, Pointer-LR, and Sequence Labeling) to understand their trade-offs between accuracy, inference speed, and training energy. The findings suggest that Biaffine parsers offer a good balance for general use, while Sequence Labeling parsers are faster for inference, though less efficient in training.

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) systems, particularly dependency parsers, are often computationally expensive, consuming significant energy for both training and inference. While accuracy is a primary concern, the increasing awareness of the environmental and practical costs of running these models necessitates a deeper understanding of their efficiency. This paper aims to provide a realistic snapshot of the efficiency landscape of contemporary dependency parsers by analyzing their performance across accuracy, inference speed, and training energy consumption, without relying on computationally heavy augmentations like pretrained language models.

Methods

The authors conducted a Pareto optimization analysis on three representative dependency parsing paradigms:

  • Deep Biaffine (BIAFFINE): A graph-based parser that uses a biaffine attention mechanism to score arc probabilities, followed by the Chu-Liu-Edmonds algorithm to construct a tree.
  • Left-to-right Pointer Network (L2R): A transition-based parser that uses a pointer network to predict attachments.
  • Sequence Labelling Parser (SEQLAB): A parser that encodes dependency trees as a sequence of labels, allowing for direct prediction per token.

They implemented these parsers using BiLSTMs as the core component and varied the number of layers and nodes to explore different model sizes. Crucially, they focused on bare-bones versions of the parsers, excluding pretrained language models or other transversal augmentations to isolate the efficiency of the core parsing mechanisms. Inference speed was measured in tokens per second on both GPU and CPU, while training energy consumption was measured in joules using the energyusage package.

Datasets

The study used four high-quality treebanks from diverse language families and subfamilies:

  • UD Hindi-HDTB (Hindi)
  • UD Polish-PDB (Polish)
  • UD Korean-Kaist (Korean)
  • Chinese Penn Treebank (Chinese)

These treebanks were chosen to balance the need for a manageable number of models with linguistic diversity and data quality.

Key Insights and Ideas

  • Biaffine as a Balanced Default: For a general balance between accuracy and speed, the Biaffine parser emerges as a strong contender.
  • SEQLAB for Speed: If inference speed is the absolute priority, the Sequence Labelling parser (SEQLAB) is the fastest option, especially for smaller models on CPU.
  • Training Efficiency Discrepancy: While accuracy and inference speed show clear trade-offs, the link between training energy and performance is less direct. SEQLAB, despite its speed advantage, is notably less efficient in terms of training energy.
  • Linearity in Training Energy: For Biaffine and SEQLAB, training energy consumption scales linearly with training time, suggesting that convergence time is the primary driver of energy cost. L2R shows a less clear but generally similar trend.
  • Model Size vs. Performance: As expected, larger networks generally lead to higher accuracy but also slower inference speeds.

Key Results

  • Inference Speed Pareto Front: The Pareto front for inference speed was primarily composed of Biaffine and SEQLAB systems. SEQLAB was consistently faster than Biaffine, especially on CPU. L2R was slower than both for comparable accuracy.
  • Training Energy Pareto Front: The analysis of training energy showed that SEQLAB was not optimal. While Biaffine and L2R showed some trade-offs, the primary factor influencing training energy was the time it took for models to converge.
  • Accuracy: The paper reports LAS scores for various configurations, with Biaffine systems generally achieving higher accuracy than SEQLAB and L2R for comparable model sizes. For instance, on the development data, Biaffine parsers with 3 layers and 1000 nodes achieved LAS scores around 86.64, while SEQLAB achieved around 81.58 and L2R around 85.93 for similar configurations.

Limitations and Future Work

The authors acknowledge several limitations:

  • Focus on BiLSTMs: The analysis primarily focused on BiLSTM parameters, acknowledging that other parameter variations could influence results.
  • Limited Treebank Diversity: While diverse, the chosen treebanks represent mainly fiction and news domains, and a wider range of domains could offer further insights.
  • Single Annotation Scheme: The study used only one dependency annotation scheme (a precursor to UD), and different schemes (e.g., SUD) might yield different results.
  • Feature Analysis: The impact of predicted POS tags, which could potentially benefit L2R and SEQLAB, was not explored.
  • Single Model Training: Only one model was trained per BiLSTM setting, and variations due to different initializations are possible, though unlikely to cause material differences.
  • Low-Resource Settings: The analysis focused on parsers trained on large annotated datasets, leaving the efficiency of low-resource parsing for future work.

Future work could involve exploring the impact of predicted POS tags, expanding the analysis to more diverse treebanks and annotation schemes, and investigating efficiency in low-resource settings.

Summary updated

BibTeX
@inproceedings{anderson2021modest,
  author = {Mark Anderson and Carlos Gómez-Rodríguez},
  title = {{A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure