On “Confirmatory” Methodological Research in Statistics and Related Fields

F. Julian D. Lange, Juliane C. Wilcke, Sabine Hoffmann, Moritz Herrmann, Anne‐Laure BoulesteixStatistics 2025 · DOI

TL;DR

This paper argues that the distinction between exploratory and confirmatory research, crucial in substantive scientific fields, is largely overlooked in statistical methodological research. The authors propose that most methodological studies are actually exploratory, even when presented as confirmatory, leading to potential misinterpretations and a lack of reliable guidance for applied researchers. They advocate for transparent reporting of research intent and the development of more rigorous, confirmatory methodological studies.

What Problem Does This Paper Address?

In many scientific disciplines, research is broadly categorized into two types: exploratory (hypothesis-generating) and confirmatory (hypothesis-testing). While both are vital, the distinction helps in interpreting results and understanding their reliability. However, in the field of statistical methodology itself – research focused on developing and evaluating statistical methods – this distinction has been largely ignored. This oversight leads to a proliferation of methods without clear guidance on their appropriate use, potentially contributing to a "replication crisis" in applied fields. Methodological researchers often present their findings as more definitive than they are, blurring the lines between generating new ideas and rigorously testing existing ones. This paper aims to bridge this gap by re-examining this distinction within methodological research and proposing ways to improve its practice and reporting.

Methods

The authors employ a two-pronged approach:

  1. Conceptual Framework: They first revisit and adapt the established definitions of exploratory and confirmatory research from substantive fields to the context of empirical methodological research. This involves clarifying what constitutes "methodological research" (studying methods using data, not just theoretical derivations) and how the exploratory/confirmatory distinction applies.
  2. Literature Survey: They conduct a survey of 115 recent articles from prominent biostatistics journals (Biometrical Journal and Statistics in Medicine). The survey extracts objective features like whether a new method is introduced, the use of terms like "exploratory" or "confirmatory," and the explicit mention of research hypotheses. The goal was not to definitively classify studies but to assess current practices and the prevalence of certain features.

Datasets

The paper does not introduce new datasets or benchmarks. The literature survey analyzed existing published articles that utilized:

  • Simulated data: This was the most common type of data used in the surveyed articles (87%).
  • Real data: Eleven articles presented real-data methodological studies.
  • Both simulated and real data: Four articles used a combination.

The paper also discusses the reuse of existing datasets in real-data methodological studies as a potential issue related to data leakage.

Key Insights and Ideas

  • The "Exploratory" Dominance: The authors strongly argue that most empirical methodological research, even when presented as confirmatory, is fundamentally exploratory. This is due to the nature of method development, where initial evaluations often lack pre-specified hypotheses and are driven by discovery.
  • Pseudo-Confirmatory Findings: A significant concern is "pseudo-confirmatory" research, where findings from exploratory work are presented as if they resulted from a confirmatory study. This can happen through HARKing (Hypothesizing After Results are Known) or other questionable research practices (QRPs).
  • Method Creation vs. Method Evaluation: Articles introducing new methods are often exploratory by nature. The process of creating a method involves trial-and-error and is inherently less neutral than evaluating an existing, well-defined method.
  • The Continuum Concept: A strict dichotomy between exploratory and confirmatory research might be too rigid. The authors suggest viewing it as a continuum, with "pure" exploration and "strict" confirmation as idealized endpoints.
  • Importance of Transparency: Transparently reporting the nature of the research (exploratory, confirmatory, or mixed) is crucial for proper interpretation and for guiding applied researchers.
  • Need for Confirmatory Methodological Research: While exploratory research is valuable for generating new ideas, there's a pressing need for more rigorous, confirmatory methodological studies to provide reliable guidance on method selection.

Key Results

The literature survey revealed:

  • Lack of Explicit Distinction: In none of the surveyed articles did authors explicitly use the terms "exploratory" or "confirmatory" to describe their research.
  • Implicit Hypotheses: While explicit hypotheses were rare (only one article explicitly stated them), many authors used wording suggesting they had expectations before analysis (e.g., "as expected").
  • Method Development Focus: A very high percentage of articles (92%) presented and evaluated a new method within the same paper, reinforcing the idea that these are often exploratory.
  • Limited Data Use in Real-Data Studies: Real-data studies often use only one or a few datasets, with significant overlap in datasets used across studies, increasing the risk of data leakage.
  • Descriptive Analysis of Simulations: Simulation studies are typically analyzed descriptively, with inferential analyses of simulation findings being rare.

Limitations and Future Work

  • Difficulty in Classification: The authors acknowledge that a clear-cut distinction between exploratory and confirmatory research is not always possible in practice, especially for mixed-methods studies.
  • Subjectivity in Assessment: While they aimed for objective features in their survey, definitively classifying studies as purely exploratory or confirmatory would inevitably involve subjectivity.
  • Need for Maturation: The field of empirical methodological research needs to "mature" to better support confirmatory studies, similar to how other fields have developed robust protocols and theoretical underpinnings.
  • Recommendations for Improvement: The paper concludes with a set of recommendations for researchers, editors, and funders to improve the reliability, reporting, and usefulness of methodological research. These include:
  • Explicitly stating the nature of the research.
  • Writing and preregistering research protocols, especially for confirmatory parts.
  • Ensuring neutrality in method evaluation.
  • Promoting standardized reporting and the sharing of code and data.
  • Encouraging more research on existing methods, particularly neutral comparison studies.
  • Funding meta-scientific projects to examine methodological research itself.

Summary updated

Abstract

Empirical substantive research, such as in the life or social sciences, is commonly categorized into the two modes exploratory and confirmatory, both of which are essential to scientific progress. The former is also referred to as hypothesis-generating or data-contingent research, while the latter is also called hypothesis-testing research. In the context of empirical methodological research in statistics, however, the exploratory-confirmatory distinction has received very little attention so far. Our paper aims to fill this gap. First, we revisit the concept of empirical methodological research through the lens of the exploratory-confirmatory distinction. Second, we examine current practice with respect to this distinction through a literature survey including 115 articles from the field of biostatistics. Third, we provide practical recommendations toward a more appropriate design, interpretation, and reporting of empirical methodological research in light of this distinction. In particular, we argue that both modes of research are crucial to methodological progress, but that most published studies-even if sometimes disguised as confirmatory-are essentially exploratory in nature. We emphasize that it may be adequate to consider empirical methodological research as a continuum between "pure" exploration and "strict" confirmation, recommend transparently reporting the mode of conducted research within the spectrum between exploratory and confirmatory, and stress the importance of study protocols written before conducting the study, especially in confirmatory methodological research.
BibTeX
@inproceedings{lange2025confirmatory,
  author = {F. Julian D. Lange and Juliane C. Wilcke and Sabine Hoffmann and Moritz Herrmann and Anne‐Laure Boulesteix},
  title = {{On “Confirmatory” Methodological Research in Statistics and Related Fields}},
  booktitle = {Statistics},
  year = {2025},
  doi = {10.1002/sim.70303},
}
Figure