Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models

Phyllis Ang, Bhuwan Dhingra, Lisa Wu Wills

Abstract

This paper investigates the trade-offs between model size, sequence length, and computational efficiency in long-context NLP models. Using the SCROLLS benchmark, we evaluate LED and Big Bird models across summarization and question-answering tasks. Our findings suggest that for summarization, increasing model size is often more energy-efficient than increasing sequence length for accuracy gains. Conversely, in question answering, smaller models excel by enabling larger training batches within fixed hardware constraints.

Introduction

The pursuit of higher accuracy in NLP has led to massive models with significant environmental costs. While benchmarks like SCROLLS measure performance on long texts, they often overlook the "Green AI" metrics—energy, speed, and power consumption. This study systematically explores "intra-model" trade-offs under fixed resource budgets (Nvidia RTX A6000 GPUs) to guide practitioners in balancing performance with computational costs.

Experimental Setup

We evaluated Longformer-Encoder-Decoder (LED-base and LED-large) and Big Bird-large architectures. Input sequence lengths were varied from 1024 to 4096 tokens. Performance was measured on four SCROLLS datasets: GovReport, SummScreenFD, and QMSum for summarization, and Qasper for question answering. Efficiency metrics included power efficiency (samples/sec/Watt), total training energy (kW-hr), and throughput.

Summarization: The "Go Big" vs. "Go Long" Trade-off

For summarization tasks, a critical finding emerged: increasing model capacity (moving from LED-base to LED-large) is frequently more energy-efficient than extending the sequence length to achieve comparable accuracy.

Figure 2: Rouge Score vs. Training Energy for summarization datasets. LED-large often achieves higher accuracy with less energy than LED-base at longer sequence lengths.

As shown in Figure 2, LED-large with a 1024-token context often outperforms LED-base with a 4096-token context while consuming less total energy. This challenges the intuition that "longer is always better" for long-context tasks, suggesting that model depth and width can sometimes compensate for a shorter context window more efficiently.

Question Answering: The Batch Size Advantage

In the Qasper QA dataset, the trend reversed. Smaller models (LED-base) proved superior in both accuracy and efficiency.

Figure 5: Efficiency vs. Accuracy (F1 Score) on the Qasper dataset. LED-base consistently outperforms larger models across various efficiency metrics.

The smaller memory footprint of LED-base allowed for significantly larger batch sizes within the fixed GPU memory budget. This increased throughput and improved learning dynamics, leading to higher F1 scores compared to LED-large and Big Bird, as illustrated in Figure 5. This highlights how hardware constraints can fundamentally shift the optimal model choice depending on the task's memory requirements.

Model Comparison and Efficiency Decay

LED consistently outperformed Big Bird in both accuracy and energy efficiency across all tested tasks. For instance, at a 4096 sequence length, Big Bird required nearly 3.5 times the training time of LED-large (48 hours vs. 14 hours). Furthermore, we observed a strong inverse correlation between sequence length and power efficiency; doubling the context window results in a more than linear increase in computational cost per token due to the underlying mechanics of attention mechanisms, even in "efficient" models.

Conclusion and Future Work

This research highlights that the optimal configuration for long-context models is highly task-dependent. We urge the NLP community to move beyond accuracy-only leaderboards and start reporting energy-per-accuracy metrics to promote more sustainable AI development. Future work should extend this analysis to newer architectures like FlashAttention-based models and even longer context windows (e.g., 32k+ tokens) to see if these trade-off patterns hold at extreme scales.

Summary updated

BibTeX
@inproceedings{ang2022characterizing,
  author = {Phyllis Ang and Bhuwan Dhingra and Lisa Wu Wills},
  title = {{Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure