Nonsense correlations in neuroscience

Kenneth D. HarrisbioRxiv 2020 · DOI

TL;DR

Neuroscience data often contains slow trends that can lead to "nonsense correlations" – statistically significant relationships between unrelated signals. This paper evaluates various methods for detecting true correlations in such data, finding that randomization and stationarity in experimental design are crucial for reliable analysis.

What Problem Does This Paper Address?

In neuroscience, researchers frequently analyze time-series data, such as neuronal firing rates and behavioral variables, to understand brain function. A common challenge is that these signals often exhibit slow, continuous trends over time. Standard statistical correlation methods, which assume data points are independent, can mistakenly identify significant correlations between completely unrelated signals when these trends align by chance. This phenomenon, termed "nonsense correlation," can lead to erroneous conclusions about neural encoding and brain-behavior relationships. The paper aims to identify robust methods for distinguishing genuine correlations from these spurious ones in the presence of such slow trends.

Methods

The authors systematically evaluated ten different methods for assessing correlations between time-series data, particularly in the context of neurophysiological signals. They simulated data with and without genuine correlations, incorporating slow drifts to mimic real-world neural and behavioral signals. The methods tested fall into several categories:

  • Pseudosession Method: This method relies on one of the time series (typically the behavioral variable) being randomly generated by the experimenter. It compares the observed association to a null distribution generated by creating many "pseudo-sessions" with random behavioral data.
  • Session Permutation Method: This approach requires multiple experimental sessions recorded under identical conditions. It tests whether neural activity predicts behavior more accurately within the same session compared to predictions made using data from different sessions.
  • Linear Shift Method: This method assumes one of the time series is stationary (its statistical properties don't change over time). It tests for correlation by comparing the association between segments of the two series when one is shifted in time. A conservative and an approximate version are discussed.
  • Other Methods: Several other methods were evaluated, including naïve correlation (standard Pearson correlation), circular shifting, phase randomization, wavelet randomization, and cross-validation (including forecasting). These methods were tested for their susceptibility to producing false positives.

Datasets

The paper does not use real-world datasets. Instead, it relies on simulated time-series data. These simulations were designed to mimic key characteristics of neurophysiological data, specifically:

  • Slowly drifting signals: Mimicking continuous trends observed in neural activity and behavior.
  • Genuine correlations: Some simulations included actual relationships between neural and behavioral variables.
  • No correlations: Other simulations generated unrelated neural and behavioral signals to test for false positives.

The simulations were run over 1000 trials for each of four scenarios: binary or continuous behavioral variables, with or without genuine correlation to neural activity.

Key Insights and Ideas

  • The ubiquity of "nonsense correlations": Standard correlation methods are highly susceptible to false positives when dealing with time-series data exhibiting slow trends, even if the signals are completely unrelated.
  • Randomization is key: Methods that incorporate randomization, either in the generation of one of the time series (pseudosession) or in the experimental design (randomized blocks/timing), are the most reliable for avoiding nonsense correlations.
  • Stationarity is a powerful assumption: If one of the time series can be assumed to be stationary, the linear shift method offers a safe and effective way to detect true correlations.
  • Many common methods fail spectacularly: Techniques like circular shifting, phase/wavelet randomization, and standard cross-validation can produce very high false-positive rates (up to 100%), making them unreliable for detecting genuine relationships in this type of data.
  • Experimental design matters most: The most robust solution is to build stationarity and randomization directly into the experimental design, enabling the use of the most reliable statistical methods.

Key Results

  • The pseudosession method and session permutation method were found to be the most reliable, producing flat distributions of p-values when no true correlation existed.
  • The linear shift method (especially the conservative version) performed well when stationarity was assumed, with low false-positive rates.
  • Methods like circular shifting, phase randomization, wavelet randomization, and standard cross-validation all showed high false-positive rates, with some reaching up to 100% in simulations where no genuine correlation was present.
  • Even forecasting cross-validation showed false positives in 61% of simulations for continuous behavioral variables.
  • Auto-decorrelation alone was not sufficient to prevent false positives, though it could reduce them when used in conjunction with other methods like the linear shift test.

Limitations and Future Work

  • Simulation-based: The findings are based on simulated data. While the simulations were designed to be representative of neurophysiological signals, real-world data may have complexities not fully captured.
  • Focus on simple correlations: The paper primarily addresses detecting simple correlations between two time series. More complex analyses, such as partial correlations in the presence of confounds, are mentioned as a harder problem requiring further investigation.
  • Experimental design recommendations: The authors strongly advocate for randomized experimental designs. This suggests future work could focus on developing and validating methods that leverage such designs more effectively.
  • Limitations of specific methods: The paper highlights that many common methods have significant limitations, implying that researchers should be cautious when applying them to time-series data with slow trends.

The paper concludes by emphasizing that building stationarity and randomization into experimental design is the most effective strategy for avoiding nonsense correlations and ensuring reliable statistical inference in neuroscience.

Summary updated

Abstract

Many neurophysiological signals exhibit slow continuous trends over time. Because standard correlation analyses assume that all samples are independent, they can yield apparently significant “nonsense correlations” even for signals that are completely unrelated. Here we compare the performance of several methods for assessing correlations between timeseries, using simulated slowly drifting signals with and without genuine correlations. The best performance was obtained from a “pseudosession method”, which relies on one of the signals being randomly generated by the experimenter, or a “session perturbation” method which requires multiple recordings under the same conditions. If neither of these is applicable, a “linear shift” method can be used when one of the signals is stationary. Methods based on cross-validation, circular shifting, phase randomization, or detrending gave up to 100% false positive rates in our simulations. We conclude that analysis of neural timeseries is best performed when stationarity and randomization is built into the experimental design.
BibTeX
@misc{harris2020nonsense,
  author = {Kenneth D. Harris},
  title = {{Nonsense correlations in neuroscience}},
  year = {2020},
  doi = {10.1101/2020.11.29.402719},
  howpublished = {bioRxiv},
}
Figure