Introduction
In the field of Natural Language Processing (NLP), benchmarks like SQuAD, GLUE, and ImageNet are traditionally viewed as neutral, scientific instruments—objective yardsticks used to measure progress. In this position paper, Rishi Bommasani challenges this technical reductionism. He argues that evaluation is not merely a "lens" through which we observe model performance, but a potent political "force" that dictates the research trajectory of the entire community.
As the field shifts toward massive Large Language Models (LLMs), a new tension has emerged: the "force of evaluation" is increasingly being overshadowed by the "force of resources" (compute and capital). This paper analyzes these power dynamics and calls for a deliberate redesign of evaluation practices to reclaim the research agenda from resource-rich entities.
Evaluation: From Technical Lens to Political Force
To understand the role of benchmarks, the paper distinguishes between two conceptual frameworks:
- Evaluation as a Lens (Technical Account): This is the standard view where evaluation provides scientific documentation, error analysis, and insight into model behavior. It treats benchmarks as passive tools for measurement.
- Evaluation as a Force (Political Account): This view recognizes evaluation as a mechanism for coordination and goal-setting. When a benchmark is widely adopted, it forces researchers to optimize for specific tasks, effectively steering the collective intellectual labor of the field.
Historical examples like ImageNet and SQuAD demonstrate this force in action. By providing a clear, shared target, these benchmarks ushered in the Deep Learning and Transformer revolutions, respectively. However, the power of an evaluation does not stem from its technical perfection; rather, it stems from social adoption. Once a benchmark becomes a standard, it gains "inertia," compelling researchers to continue using it—even when flawed—to ensure head-to-head comparability with prior work.
The Conflict: Evaluation vs. Resources
Currently, the field is witnessing a shift in who holds the power to set the research agenda. Historically, well-designed evaluations could drive the field forward regardless of a lab's size. Today, however, the "force of resources"—the sheer ability to scale compute and data—has become the primary driver of progress.
This shift has several concerning implications:
- The Waning Influence of Benchmarks: In the LLM era, models are often released with inconsistent or ad-hoc evaluations. The paper notes that across 32 major LLM papers, Recognizing Textual Entailment (RTE) was the only dataset evaluated in more than 50% of them. This lack of coordination suggests that the "force of resources" is currently winning the battle for the field's direction.
- The Cost of Judging: Evaluation itself is becoming a resource-intensive activity. For instance, a single run of the HELM (Holistic Evaluation of Language Models) framework costs approximately $40,000 USD and 20,000 GPU hours. This creates a significant barrier to entry, where only elite academic and corporate labs can afford to participate in high-status evaluation.
- Legitimacy and Values: While adoption of a benchmark is technically voluntary (consensual), the creation of these benchmarks is increasingly concentrated in a few powerful institutions. This raises critical questions about whose values—accuracy, efficiency, fairness, or safety—are being imposed on the global research community.
Reclaiming the Agenda through Pluralism
Despite the current dominance of resources, evaluation remains the best tool for promoting pluralism (diversity) in research. Unlike scaling laws, which prioritize raw accuracy and size, evaluation frameworks can be intentionally designed to reward a variety of virtues, such as robustness, uncertainty quantification, and fairness.
To reclaim the research agenda, the author suggests several paths forward:
- Aggressive Coordination: The community must intentionally harness evaluation to drive change rather than letting it happen as a byproduct of resource availability.
- Elevating Evaluation Status: Contributions to datasets and benchmarks (such as the NeurIPS Datasets and Benchmarks track) must be recognized as high-status research on par with algorithmic innovation.
- Community-Driven Design: Supporting decentralized, community-driven evaluation efforts can help ensure that the metrics used to judge models reflect a broader set of societal values rather than just the interests of resource-rich actors.
Conclusion
Evaluation is a site of power. While the "force of resources" currently dictates much of the NLP landscape, the intentional design and adoption of benchmarks offer a pathway to a more diverse and democratized future. By treating evaluation as a political tool for coordination rather than just a technical metric, the research community can steer the field toward goals that matter beyond mere scale.
Summary updated