Abstract
This paper investigates the trade-offs between model size, sequence length, and computational efficiency in long-context NLP models. Using the SCROLLS benchmark, we evaluate LED and Big Bird models across summarization and question-answering tasks. Our findings suggest that for summarization, increasing model size is often more energy-efficient than increasing sequence length for accuracy gains. Conversely, in question answering, smaller models excel by enabling larger training batches within fixed hardware constraints.
Introduction
The pursuit of higher accuracy in NLP has led to massive models with significant environmental costs. While benchmarks like SCROLLS measure performance on long texts, they often overlook the "Green AI" metrics—energy, speed, and power consumption. This study systematically explores "intra-model" trade-offs under fixed resource budgets (Nvidia RTX A6000 GPUs) to guide practitioners in balancing performance with computational costs.
Experimental Setup
We evaluated Longformer-Encoder-Decoder (LED-base and LED-large) and Big Bird-large architectures. Input sequence lengths were varied from 1024 to 4096 tokens. Performance was measured on four SCROLLS datasets: GovReport, SummScreenFD, and QMSum for summarization, and Qasper for question answering. Efficiency metrics included power efficiency (samples/sec/Watt), total training energy (kW-hr), and throughput.
Summarization: The "Go Big" vs. "Go Long" Trade-off
For summarization tasks, a critical finding emerged: increasing model capacity (moving from LED-base to LED-large) is frequently more energy-efficient than extending the sequence length to achieve comparable accuracy.
As shown in Figure 2, LED-large with a 1024-token context often outperforms LED-base with a 4096-token context while consuming less total energy. This challenges the intuition that "longer is always better" for long-context tasks, suggesting that model depth and width can sometimes compensate for a shorter context window more efficiently.
Question Answering: The Batch Size Advantage
In the Qasper QA dataset, the trend reversed. Smaller models (LED-base) proved superior in both accuracy and efficiency.
The smaller memory footprint of LED-base allowed for significantly larger batch sizes within the fixed GPU memory budget. This increased throughput and improved learning dynamics, leading to higher F1 scores compared to LED-large and Big Bird, as illustrated in Figure 5. This highlights how hardware constraints can fundamentally shift the optimal model choice depending on the task's memory requirements.
Model Comparison and Efficiency Decay
LED consistently outperformed Big Bird in both accuracy and energy efficiency across all tested tasks. For instance, at a 4096 sequence length, Big Bird required nearly 3.5 times the training time of LED-large (48 hours vs. 14 hours). Furthermore, we observed a strong inverse correlation between sequence length and power efficiency; doubling the context window results in a more than linear increase in computational cost per token due to the underlying mechanics of attention mechanisms, even in "efficient" models.
Conclusion and Future Work
This research highlights that the optimal configuration for long-context models is highly task-dependent. We urge the NLP community to move beyond accuracy-only leaderboards and start reporting energy-per-accuracy metrics to promote more sustainable AI development. Future work should extend this analysis to newer architectures like FlashAttention-based models and even longer context windows (e.g., 32k+ tokens) to see if these trade-off patterns hold at extreme scales.
Summary updated