Is More Data All You Need? A Causal Exploration

Athanasios Vlontzos, Hadrien Reynaud, Bernhard KainzarXiv 2022 · DOI · arXiv

Introduction

In medical AI, data acquisition is a significant bottleneck. Curating high-quality, annotated datasets for tasks such as diagnosing Diabetic Retinopathy requires expert clinical time and substantial resources. When a model fails on specific patient types or disease classes, the standard industry response is often a vague mandate to "collect more data." However, this approach lacks precision: it does not specify how much more data is required or which classes should be prioritized. This paper introduces a principled, causal framework to reason about dataset composition and its direct effect on model performance, moving beyond simple uncertainty-based heuristics.

Framework: Structural Causal Models

The researchers shift the focus from statistical correlation to Structural Causal Models (SCMs) and Counterfactual Inference. By treating the training dataset as a "treatment" variable ($X$) and the model's prediction as the "outcome" ($Y$), they can quantify the "Probability of Sufficiency."

Structural Causal Model representing the training process as a treatment intervention.

The core methodology follows the Abduction-Action-Prediction paradigm to answer a specific counterfactual: "Given that this sample was misclassified by the current model, would it have been correct if the model had been trained on a different dataset?" 1. Abduction: Infer the latent state of the system based on the current model's failure. 2. Action: Use the "do-operator" to simulate a dataset intervention (e.g., increasing size to 1,000 samples). 3. Prediction: Calculate the probability that the sample's classification "flips" from incorrect to correct under the new regime.

Experimental Setup

The framework was evaluated using two distinct datasets:

  • MorphoMNIST: A synthetic version of MNIST allowing for controlled morphological perturbations (e.g., fractures), providing a ground-truth environment for simulating new data gathering regimes.
  • Kaggle Diabetic Retinopathy (DR): A real-world medical dataset focused on "No DR," "Mild," and "Moderate" cases to explore the impact of class imbalance on clinical diagnostics.

Analysis of Data Interventions

The study reveals that the "Law of Diminishing Returns" applies heavily to random data collection. In the synthetic MorphoMNIST trials, increasing a base dataset from 5,000 to 10,000 samples yielded only a 9% chance of correcting a previously misclassified sample.

Probability of a misclassified sample 'flipping' to correct when increasing dataset size (Synthetic Data).

As shown in the results for synthetic data, moving from a very small dataset (100 samples) to a larger one (1,000 samples) offers a 23.17% chance of correcting a failure. However, as the baseline size increases, the probability of a "flip" decreases significantly. Conversely, "informed interventions"—where the specific class the model is failing on is upsampled—act as force multipliers. These targeted additions resulted in a significantly higher probability of correcting errors (averaging ~67%) compared to random additions.

In the real-world Diabetic Retinopathy dataset, the authors found that adding "Moderate DR" samples improved performance across all initial dataset sizes. However, randomly increasing a medical dataset from 100 to 2,000 samples only provided a ~10% chance of a sample flipping from wrong to right, highlighting the inefficiency of unguided collection.

Conclusion and Future Work

Causal analysis provides a pathway toward "data minimization," allowing practitioners to determine the minimum amount of data required to reach specific performance thresholds. This is particularly relevant for regulatory compliance (e.g., GDPR) and resource optimization in clinical settings.

A current limitation is the requirement for a well-defined causal graph (DAG). Future research will focus on estimating these causal effects when the underlying data generation process is not fully known and integrating these "confidence thresholds" into the official approval processes for medical AI devices.

Summary updated

Abstract

Curating a large scale medical imaging dataset for machine learning applications is both time consuming and expensive. Balancing the workload between model development, data collection and annotations is difficult for machine learning practitioners, especially under time constraints. Causal analysis is often used in medicine and economics to gain insights about the effects of actions and policies. In this paper we explore the effect of dataset interventions on the output of image classification models. Through a causal approach we investigate the effects of the quantity and type of data we need to incorporate in a dataset to achieve better performance for specific subtasks. The main goal of this paper is to highlight the potential of causal analysis as a tool for resource optimization for developing medical imaging ML applications. We explore this concept with a synthetic dataset and an exemplary use-case for Diabetic Retinopathy image analysis.
BibTeX
@misc{vlontzos2022is,
  author = {Athanasios Vlontzos and Hadrien Reynaud and Bernhard Kainz},
  title = {{Is More Data All You Need? A Causal Exploration}},
  year = {2022},
  eprint = {2206.02409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.02409},
}
Figure