Time to Stop and Think: What kind of research do we want to do?

Josu Ceberio, Borja CalvoarXiv 2024 · DOI · arXiv

Abstract

In the field of metaheuristic optimization, a growing trend of "inertial behavior" threatens the scientific and practical utility of research. This paper, based on the work of Josu Ceberio and Borja Calvo, critiques the current culture of chasing marginal gains on artificial benchmarks and proposes a clear distinction between engineering-driven and science-driven research. By aligning methodology with specific goals—either solving real-world problems or gaining fundamental understanding—the community can move beyond the "publish or perish" cycle toward more meaningful contributions.

Introduction: The Inertial Trap

The field of metaheuristic optimization currently finds itself at a crossroads. While the volume of publications continues to grow, there is a mounting concern that the quality and impact of this research are stagnating. This "inertial behavior" is characterized by a rote adherence to tradition: researchers use specific statistical tests, such as $p$-values ($p < 0.05$), and target established benchmarks not because they are the most appropriate tools, but because they are what the community—and by extension, journal editors—expects.

This cycle is fueled by a "publish or perish" culture that prioritizes quantity over substance. The result is a "reproducibility crisis" where many proposed algorithms offer only marginal, often hand-picked, improvements over the state-of-the-art (SOTA) without providing any deeper insight into why the improvement occurred or how it translates to real-world utility.

A Conceptual Framework: Engineering vs. Science

To break this cycle, a fundamental shift in how we categorize and conduct research is required. The authors propose a clear distinction between two primary research motivations: the Engineering Approach and the Scientific Approach.

The Engineering Approach

The Engineering Approach is fundamentally utilitarian. Its primary objective is to solve a specific, real-world optimization problem for a stakeholder, such as an industrial partner or a public institution. In this context, success is not measured by performance on an abstract benchmark, but by the utility and efficiency of the solution in its intended environment. Research in this vein must be validated against the actual constraints and requirements of the "customer," rather than artificial metrics.

The Scientific Approach

Conversely, the Scientific Approach is knowledge-driven. Its goal is to understand the underlying behavior of algorithms or the structural characteristics of optimization problems. This approach follows the classical scientific method: observing phenomena, formulating hypotheses, and conducting rigorous, controlled experiments to validate or refute those hypotheses. Here, a "failed" algorithm—one that does not beat the SOTA—is still a valuable scientific contribution if it provides insight into the limitations of a specific technique or the nature of a problem space.

The SOTA Obsession and the Misuse of Benchmarks

One of the most significant hurdles to progress is the obsessive search for algorithms that universally outperform existing methods. This pursuit is often misleading. The "No Free Lunch" theorems suggest that no single algorithm can be superior across all possible problems. Therefore, claiming universal superiority by hand-picking instances where a new method happens to excel is scientifically disingenuous.

Furthermore, the reliance on artificial benchmarks (e.g., standard Traveling Salesperson Problem or Graph Coloring instances) often creates a gap between research and reality. While these benchmarks are useful for controlled scientific inquiry, they are frequently used as proxies for real-world performance, which they are not. Engineering-driven research should focus on real-world instances, while scientific research should use benchmarks to probe specific algorithmic properties rather than just "winning" a leaderboard.

Parallelism with Natural Sciences

As optimization problems grow in complexity, the authors suggest that the field should adopt methodologies from the natural sciences. Just as biologists or physicists use experimental science to study systems too complex for pure mathematical derivation, optimization researchers should treat their algorithms as complex systems. This involves moving beyond simple performance comparisons and toward a more granular understanding of how different algorithmic components interact with problem features.

Conclusion: A Shared Reflection

The current state of metaheuristic research is a systemic issue that cannot be solved by individual researchers alone. It requires a collective reflection on what makes research "worthy" of publication. We must move away from the circular logic of doing research "the way it has always been done" and instead demand clarity: Is a paper solving a real problem (Engineering), or is it expanding our understanding of optimization (Science)?

The future of the field depends on our ability to communicate results more effectively and to value the discovery of "why" just as much as the achievement of "how fast." By choosing a clear path, researchers can ensure their work contributes to a meaningful body of knowledge rather than merely adding to the noise of inertial publication.

Summary updated

Abstract

Experimentation is an intrinsic part of research in artificial intelligence since it allows for collecting quantitative observations, validating hypotheses, and providing evidence for their reformulation. For that reason, experimentation must be coherent with the purposes of the research, properly addressing the relevant questions in each case. Unfortunately, the literature is full of works whose experimentation is neither rigorous nor convincing, oftentimes designed to support prior beliefs rather than answering the relevant research questions. In this paper, we focus on the field of metaheuristic optimization, since it is our main field of work, and it is where we have observed the misconduct that has motivated this letter. Even if we limit the focus of this manuscript to the experimental part of the research, our main goal is to sew the seed of sincere critical assessment of our work, sparking a reflection process both at the individual and the community level. Such a reflection process is too complex and extensive to be tackled as a whole. Therefore, to bring our feet to the ground, we will include in this document our reflections about the role of experimentation in our work, discussing topics such as the use of benchmark instances vs instance generators, or the statistical assessment of empirical results. That is, all the statements included in this document are personal views and opinions, which can be shared by others or not. Certainly, having different points of view is the basis to establish a good discussion process.
BibTeX
@misc{ceberio2024time,
  author = {Josu Ceberio and Borja Calvo},
  title = {{Time to Stop and Think: What kind of research do we want to do?}},
  year = {2024},
  eprint = {2402.08298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.08298},
}
Figure