Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models

Phyllis Ang, Bhuwan Dhingra, Lisa Wu Wills

TL;DR

This paper systematically investigates the trade-off between accuracy and efficiency (energy, speed) for long-context NLP models like Longformer-Encoder-Decoder (LED) and Big Bird. They find that for summarization, larger models are more energy-efficient than longer sequences, while for question answering, smaller models with larger batch sizes are more efficient and accurate.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen remarkable progress, often driven by larger and more complex models that achieve higher accuracy. However, this progress comes at a significant computational and energy cost, raising concerns about the environmental impact of AI research (the "Green AI" initiative). While new benchmarks like SCROLLS are emerging to evaluate long-context NLP models, they primarily focus on accuracy and often overlook efficiency metrics like speed and energy consumption. Furthermore, existing efficiency analyses typically compare different model architectures with fixed hyperparameters. This paper addresses the crucial question of how to navigate the accuracy-efficiency trade-off within a single model architecture by varying hyperparameters like model size and input sequence length, especially under resource constraints. Understanding this trade-off is vital for practitioners to make informed decisions about model selection and hyperparameter tuning.

Methods

The researchers conducted a systematic study on two prominent long-context encoder-decoder NLP models: Longformer-Encoder-Decoder (LED) and Big Bird. They fine-tuned and evaluated these models on four datasets from the SCROLLS benchmark, which are designed for tasks requiring the processing of long sequences.

The core of their methodology involved:

  • Varying Hyperparameters: They explored the impact of two key hyperparameters:
  • Sequence Length: Testing lengths of 1024, 2048, 3072, and 4096 tokens.
  • Model Size: Comparing "base" and "large" versions of LED, and only the "large" version of Big Bird (as a base version wasn't readily available).
  • Resource Constraints: To simulate realistic scenarios, they used a fixed resource budget:
  • For summarization tasks, a single Nvidia RTX A6000 GPU was used.
  • For question answering, a more substantial budget of 8 Nvidia RTX A6000 GPUs was employed to allow for larger batch sizes, which were found to be critical for this task.
  • Efficiency Metrics: They measured efficiency using:
  • Total energy consumption during training.
  • Training speed (samples per second).
  • Inference speed (samples per second).
  • Power efficiency (samples per second per Watt).
  • Accuracy Metrics: Standard metrics for the tasks were used: Rouge scores for summarization and F1 scores for question answering.

Datasets

The study utilized four datasets from the SCROLLS benchmark:

  • GovReport: A summarization dataset of government reports.
  • SummScreenFD: A summarization dataset of TV show transcripts.
  • QMSum: A query-based summarization dataset of meeting notes.
  • Qasper: A question answering dataset focused on NLP papers.

These datasets are specifically chosen because they naturally involve long input sequences, making them suitable for evaluating long-context models.

Key Insights and Ideas

  • Summarization: Model Size > Sequence Length for Energy Efficiency: Surprisingly, increasing model size (e.g., from LED-base to LED-large) was found to be a more energy-efficient way to achieve higher accuracy in summarization compared to simply increasing the input sequence length. This suggests that for summarization, optimizing model architecture might be more impactful for energy savings than pushing sequence length limits, especially if inference speed isn't the primary concern.
  • Question Answering: Smaller Models Shine with Larger Batches: For question answering, smaller models (like LED-base) were found to be both more efficient and more accurate. This is attributed to the ability to fit larger training batch sizes on the available hardware, which significantly boosts performance on this task.
  • Big Bird vs. LED Trade-offs: LED consistently achieved better accuracy at lower energy costs than Big Bird across the summarization tasks. While Big Bird's power efficiency was comparable, its Rouge scores were lower.
  • Sequence Length Impact on Speed: As input sequence lengths increase, both training and inference speeds decrease for these Transformer models, a consequence of their sub-quadratic attention mechanisms.
  • GPU Utilization vs. Training Time: Average GPU utilization alone is not a reliable predictor of energy consumption. Training time proved to be a more indicative metric, as energy is power consumed over time.

Key Results

  • Summarization:
    • LED models generally outperformed Big Bird in terms of accuracy and energy efficiency.
  • Increasing model size for summarization led to better accuracy with less energy cost than increasing sequence length.
  • Inference speed decreased significantly with longer sequences, but smaller models were faster at the cost of accuracy.
  • Question Answering (Qasper):
    • LED-base models were more efficient and accurate than LED-large models.
  • Larger batch sizes (enabled by using more GPUs and gradient accumulation) were crucial for achieving higher F1 scores.
  • Increasing sequence length provided significant accuracy gains but at the cost of slower inference speed.
  • For Qasper with input length 4096, the LED-base model achieved an F1 score ~10 points higher than reported in the original SCROLLS paper.

Limitations and Future Work

  • Limited Model Scope: The study focused on only two specific long-context models (LED and Big Bird). The findings might not generalize to all long-context architectures.
  • Big Bird Base Model: The analysis for Big Bird was limited to the large version due to the unavailability of a readily accessible base model.
  • Hyperparameter Tuning: While they explored sequence length and model size, other hyperparameters could also influence the trade-off.
  • Future Work: The authors suggest that future work could extend this analysis to more models and explore the impact of other hyperparameters. They also highlight the importance of integrating efficiency metrics into future NLP benchmarks to encourage the development of more sustainable AI models.

Summary updated

BibTeX
@inproceedings{ang2022characterizing,
  author = {Phyllis Ang and Bhuwan Dhingra and Lisa Wu Wills},
  title = {{Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure