Consensus on the reporting and experimental design of clinical and cognitive-behavioural neurofeedback studies (CRED-nf checklist)

Tomas Ros, Stefanie Enriquez‐Geppert, Vadim Zotev, Kymberly D. Young, Guilherme Wood, Susan Whitfield‐Gabrieli, Feng Wan, François Vialatte, Dimitri Van De Ville, Doron Todder, Tanju Sürmeli, James Sulzer, Ute Strehl, Barry Sterman, Naomi J. Steiner, Bettina Sorger, Surjo R. Soekadar, Ranganatha Sitaram, Leslie Sherlin, Michael Schönenberg, Frank Scharnowski, Manuel Schabus, Katya Rubia, Alessandro Pasquale De Rosa, Miriam Reiners, Jaime Pineda, Christian Paret, Andrew G. Nicholson, Wenya Nan, Javier Mínguez, Jean‐Arthur Micoulaud‐Franchi, David Marc Anton Mehler, Joel F. Lubar, Fabien Lotte, David E.j. Linden, Jarrod A. Lewis‐Peacock, Mikhail Lebedev, Ruth A. Lanius, Andrea Kübler, Cornelia Kranczioch, Yury Koush, Lilian Konicar, Silivia E. Kober, Manousos A. Klados, Camille Jeunet, Tieme W. P. Janssen, René J. Huster, Kerstin Hoedlmoser, Laurence Hirshberg, Talma Hendler, Michelle Hampson, Adrian G. Guggisberg, John Gruzelier, Rainer Göbel, Nicolas Gninenko, Alireza Gharabaghi, Paul Frewen, Thomas Fovet, Thalı́a Fernández, Carlos Escolano, Ann-Christine Ehl, Renate Drechsler, R. Christopher deCharms, Dirk De Ridder, Eddy J. Davelaar, Marco Congedo, Marc Cavazza, Rien M. H. M. Breteler, Daniel Brandeis, Jerzy Bodurka, Niels Birbaumer, Olga Bazanova, Robert Guggenberger, Beatrix Barth, Panagiotis D. Bamidis, Tibor Auer, Martijn Arns, Robert T. ThibaultPsyArXiv 2019 · DOI

TL;DR

Neurofeedback research is booming, but a lack of standardized reporting and experimental design makes it hard to compare studies and draw firm conclusions. This paper introduces the CRED-nf checklist, a consensus-driven framework to improve the rigor and clarity of neurofeedback research, ultimately accelerating our understanding of its mechanisms and efficacy.

What Problem Does This Paper Address?

Neurofeedback, a technique where individuals learn to regulate their own brain activity, has garnered significant attention in both scientific and clinical communities. However, the field faces a critical challenge: a lack of standardized reporting and experimental design. This variability makes it difficult to interpret findings, replicate studies, and definitively attribute observed effects to neurofeedback-specific mechanisms versus non-specific factors (like placebo effects, participant motivation, or general cognitive training). The authors highlight a debate about whether neurofeedback truly offers unique benefits beyond these non-specific influences. To address this, they propose a comprehensive checklist to guide researchers in designing and reporting their studies more rigorously, fostering a more robust and reproducible evidence base.

Methods

The core contribution of this paper is the development and presentation of the CRED-nf checklist. This checklist is not a novel experimental method itself, but rather a set of guidelines for how to conduct and report neurofeedback research. The authors, a group of neurofeedback researchers, arrived at this consensus through an iterative process involving adversarial collaboration, feedback from researchers involved in recent correspondences, and sharing with the broader neurofeedback community. The checklist is structured into domains such as "Pre-experiment," "Control groups," "Control measures," "Feedback specifications," "Outcome measures (Brain and Behaviour)," and "Data storage." Each item within these domains is categorized as either "essential" or "encouraged," with clear explanations and examples from published literature. The checklist is designed to be used in conjunction with an online tool and aims to complement existing reporting guidelines like CONSORT.

Datasets

Not discussed in the paper. The paper focuses on establishing reporting standards rather than presenting new experimental results based on specific datasets.

Key Insights and Ideas

  • The "Multi-Component Model" of Neurofeedback: The paper visualizes how the effects of neurofeedback are likely driven by a combination of factors, including neurofeedback-specific mechanisms, neurofeedback-unique non-specific factors (e.g., the interaction with the trainer), general non-specific factors (placebo, motivation), and repetition-related effects. Understanding the contribution of each is crucial.
  • The Need for Rigorous Control: A major theme is the importance of well-designed control groups and conditions to disentangle neurofeedback-specific effects from non-specific ones. This includes placebo controls (sham neurofeedback) and active controls.
  • Blinding is Crucial (and Difficult): The checklist emphasizes the importance of blinding participants, researchers, and statisticians where possible, acknowledging the inherent challenges in blinding neurofeedback studies.
  • Beyond Brain Activity: The paper stresses the need to measure clinically and behaviorally significant outcomes, not just changes in brain activity, and to define these a priori.
  • Transparency in Data and Analysis: The encouragement to upload all materials, analysis scripts, code, and raw data to open-access repositories promotes reproducibility and transparency.

Key Results

The primary "result" of this paper is the CRED-nf checklist itself. It's not a quantitative finding but a qualitative framework. The paper provides numerous examples from existing literature to illustrate each checklist item, demonstrating how current research often falls short of ideal reporting standards. The checklist aims to guide future research towards:

  • Improved Experimental Design: Encouraging preregistration, appropriate sample size justification, and the use of robust control groups.
  • Enhanced Reporting Clarity: Mandating detailed descriptions of feedback modalities, feature extraction, reinforcement schedules, and outcome measures.
  • Increased Reproducibility: Promoting transparency in data and analysis sharing.

Limitations and Future Work

The authors acknowledge that the checklist is a "first iteration" and invite community feedback for improvement. They also note that currently, few neurofeedback software packages are designed for blinding treatment staff, which is a practical limitation for implementing some checklist items. Future work will involve the ongoing refinement of the checklist based on community input and its widespread adoption by researchers and journals to elevate the quality of neurofeedback research.

Summary updated

Abstract

This checklist is intended to encourage robust experimental design and clear reporting for clinical and cognitive-behavioural neurofeedback experiments.
BibTeX
@misc{ros2019consensus,
  author = {Tomas Ros and Stefanie Enriquez‐Geppert and Vadim Zotev and Kymberly D. Young and Guilherme Wood and Susan Whitfield‐Gabrieli and Feng Wan and François Vialatte and Dimitri Van De Ville and Doron Todder and Tanju Sürmeli and James Sulzer and Ute Strehl and Barry Sterman and Naomi J. Steiner and Bettina Sorger and Surjo R. Soekadar and Ranganatha Sitaram and Leslie Sherlin and Michael Schönenberg and Frank Scharnowski and Manuel Schabus and Katya Rubia and Alessandro Pasquale De Rosa and Miriam Reiners and Jaime Pineda and Christian Paret and Andrew G. Nicholson and Wenya Nan and Javier Mínguez and Jean‐Arthur Micoulaud‐Franchi and David Marc Anton Mehler and Joel F. Lubar and Fabien Lotte and David E.J. Linden and Jarrod A. Lewis‐Peacock and Mikhail Lebedev and Ruth A. Lanius and Andrea Kübler and Cornelia Kranczioch and Yury Koush and Lilian Konicar and Silivia E. Kober and Manousos A. Klados and Camille Jeunet and Tieme W. P. Janssen and René J. Huster and Kerstin Hoedlmoser and Laurence Hirshberg and Talma Hendler and Michelle Hampson and Adrian G. Guggisberg and John Gruzelier and Rainer Göbel and Nicolas Gninenko and Alireza Gharabaghi and Paul Frewen and Thomas Fovet and Thalı́a Fernández and Carlos Escolano and Ann-Christine Ehl and Renate Drechsler and R. Christopher deCharms and Dirk De Ridder and Eddy J. Davelaar and Marco Congedo and Marc Cavazza and Rien M. H. M. Breteler and Daniel Brandeis and Jerzy Bodurka and Niels Birbaumer and Olga Bazanova and Robert Guggenberger and Beatrix Barth and Panagiotis D. Bamidis and Tibor Auer and Martijn Arns and Robert T. Thibault},
  title = {{Consensus on the reporting and experimental design of clinical and cognitive-behavioural neurofeedback studies (CRED-nf checklist)}},
  year = {2019},
  doi = {10.31234/osf.io/nyx84},
  howpublished = {PsyArXiv},
}
Figure