How much does your data exploration overfit? Controlling bias via information usage

Daniel Russo, James Zou

TL;DR

This paper introduces a novel information-theoretic framework to quantify and bound the bias introduced by data exploration. It provides a unified lens to understand how adaptive analysis, where decisions are made based on observed data, can lead to inflated results and false discoveries, offering a way to control this "researcher degrees of freedom."

What Problem Does This Paper Address?

In modern data science, analysts often don't know the right questions to ask a priori. Instead, they engage in an adaptive process of exploratory data analysis (EDA), where initial observations inform subsequent analyses. This "peeking" at the data, even with good intentions, invalidates standard statistical theory, which assumes hypotheses are pre-specified. This adaptive process can lead to "researcher degrees of freedom," a form of overfitting where the reported results are influenced by the data itself, leading to inflated estimates and false discoveries. While it's known that different EDA strategies have varying risks, there's a lack of a formal framework to quantify and compare this bias. This paper aims to bridge this gap by providing a principled way to measure and control exploration bias.

Methods

The core of the paper is a general framework based on mutual information to quantify "information usage." Specifically, they define information usage as the mutual information between the analyst's selection rule (which analysis to report) and the actual data statistics. They prove that this information usage provides an upper bound on the bias of the reported estimate. This framework allows them to analyze various EDA procedures, including filtering, rank selection, and clustering. They also explore how randomization techniques can be used to reduce information usage and, consequently, bias. The paper also extends these bounds to other error metrics like absolute and squared error, and discusses connections to differential privacy and algorithmic stability.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and uses illustrative examples and simulations rather than specific datasets.

Key Insights and Ideas

  • Information Usage as a Bias Metric: The paper establishes that the mutual information between the selection rule and the data statistics (information usage) is a direct measure of exploration bias. Higher information usage implies a greater risk of overfitting.
  • Signal Reduces Bias: The presence of a strong "signal" in the data (i.e., a clear underlying pattern) can naturally reduce the information usage and thus the bias of exploratory analyses, making them more reliable.
  • Randomization as a Bias Control: Introducing controlled randomness into the data exploration process can reduce information usage and bias, even when the analyst's strategy is complex or unknown. This is demonstrated through randomized selection schemes.
  • Unified Framework: The information usage framework provides a single lens to analyze and compare the bias introduced by diverse EDA procedures, offering a more principled approach than ad-hoc methods.

Key Results

  • Upper Bounds on Bias: Proposition 1 provides a general bound on the bias of an arbitrary adaptive data analysis procedure, showing it's proportional to the square root of the information usage.
  • Tightness of Bounds: The paper demonstrates that these bounds are tight in certain settings, such as max-selection of Gaussian variables, where information usage precisely quantifies the bias.
  • Analysis of Specific Procedures: The framework is applied to analyze the bias introduced by filtering, rank selection, and visualization techniques, providing insights into when these methods are more or less prone to bias.
  • Randomization Benefits: Simulations show that randomized selection (e.g., max-entropy selection) can significantly decrease bias compared to deterministic rank selection, especially in low-signal regimes, albeit sometimes at the cost of accuracy.

Limitations and Future Work

  • Data-Dependent Bounds: While a strength, the data-dependent nature of the bounds means they can be loose in some scenarios. The authors suggest exploring more general exploration procedures where information usage provides a tighter approximation to bias.
  • Practical Implementation: While the paper motivates randomization, further research into practical implementations of these randomization approaches in real-world analytic settings is suggested.
  • Combining Approaches: The authors highlight the potential for combining the sharpness of post-selection inference and differential privacy with the generality of their information usage framework.
  • Worst-Case vs. Average-Case: The framework provides bounds on average bias. Future work could explore worst-case guarantees, potentially by leveraging concepts like max-information.

Summary updated

Abstract

Modern data is messy and high-dimensional, and it is often not clear a priori what are the right questions to ask. Instead, the analyst typically needs to use the data to search for interesting analyses to perform and hypotheses to test. This is an adaptive process, where the choice of analysis to be performed next depends on the results of the previous analyses on the same data. Ultimately, which results are reported can be heavily influenced by the data. It is widely recognized that this process, even if well-intentioned, can lead to biases and false discoveries, contributing to the crisis of reproducibility in science. But while any data-exploration renders standard statistical theory invalid, experience suggests that different types of exploratory analysis can lead to disparate levels of bias, and the degree of bias also depends on the particulars of the data set. In this paper, we propose a general information usage framework to quantify and provably bound the bias and other error metrics of an arbitrary exploratory analysis. We prove that our mutual information based bound is tight in natural settings, and then use it to give rigorous insights into when commonly used procedures do or do not lead to substantially biased estimation. Through the lens of information usage, we analyze the bias of specific exploration procedures such as filtering, rank selection and clustering. Our general framework also naturally motivates randomization techniques that provably reduce exploration bias while preserving the utility of the data analysis. We discuss the connections between our approach and related ideas from differential privacy and blinded data analysis, and supplement our results with illustrative simulations.
BibTeX
@inproceedings{russo2016how,
  author = {Daniel Russo and James Zou},
  title = {{How much does your data exploration overfit? Controlling bias via information usage}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2016},
}
Figure