Probing the Effect of Selection Bias on Generalization: A Thought Experiment

John K. Tsotsos, Jun Luo

TL;DR

This paper argues that current deep learning models might overstate their generalization capabilities due to biases in training data. Using a thought experiment with LEGO bricks, they demonstrate that even with a perfect model and zero training error, missing key data dimensions (like color or pose) can lead to catastrophic generalization failures, highlighting the critical need to consider dataset completeness and performance specifications.

What Problem Does This Paper Address?

The paper tackles a fundamental question in machine learning: how well do models truly generalize when trained on datasets that are, by necessity, incomplete samples of the real world? While deep learning models often achieve impressive performance, the authors contend that the training data used is typically orders of magnitude smaller than the full population of possible data. This small sampling inherently introduces biases, and the paper questions the limits of generalization when these biases are present. Specifically, they ask: "If a network is trained with a dataset that misses particular values of some defining domain attribute, can it generalize to the full domain from which that training dataset was extracted while maintaining its performance accuracy?" This is crucial because many real-world applications, especially safety-critical ones, demand robust generalization, and current methods for validating training data are often insufficient.

Methods

The authors employ a "Thought Experiment" approach, specifically using Reductio ad Absurdum via Backcasting. This involves:

  1. Defining a Hypothetical Future: Assuming a perfect recognition system for a specific task (LEGO brick binning) that achieves extremely low error (less than LEGO's manufacturing error).
  2. Introducing Bias: Manipulating the training dataset by systematically removing specific attribute values (e.g., a particular color, size, pose, or shape).
  3. Backcasting: Working backward from the assumed perfect future to see if the biased training can still lead to the desired performance. If it fails to meet the performance specification, it serves as a proof by contradiction that the initial speculation (that generalization overcomes bias) is flawed.

This method allows them to probe generalization limits without needing to build and test actual complex systems, focusing instead on the theoretical impact of specific data omissions.

Datasets

The paper does not use traditional datasets for training or evaluation in the typical sense. Instead, it defines a hypothetical domain based on:

  • LEGO® Bricks: A well-defined, cataloged domain with a known number of pieces, colors, sizes, and poses. The authors use data from online sources (like Boen, 2018) to characterize the dimensionality of this domain, estimating the total number of unique LEGO® pieces to be around 73,320 (520 brick types x 141 colors).
  • Hypothetical Training/Test Sets: These are constructed by taking the "full set of samples" (as defined by the LEGO® domain's characteristics) and then systematically removing specific attribute values to simulate selection bias.

Key Insights and Ideas

  • Generalization is Not Magic: The paper challenges the implicit assumption that deep learning models inherently bridge data gaps. It suggests that generalization is not a free pass but is heavily dependent on how well the training data represents the defining dimensions of the domain.
  • Performance Specifications are Paramount: The utility of generalization is directly tied to the acceptable error rate for a given application. A model might appear to generalize well on some metrics, but if it fails to meet critical performance thresholds, that generalization is useless.
  • "Left Out" Dimensions Can Be Irrecoverable: Specific visual features (like color or 3D pose) that are not sufficiently represented in training data may not be recoverable through generalization alone. The model might learn to approximate, but not perfectly replicate, these missing aspects.
  • Bias is Hidden and Silent: The authors emphasize that selection bias can be subtle and difficult to detect. Without rigorous analysis of training data representativeness, systems might appear to perform well until encountering specific, under-represented scenarios.
  • Thought Experiments as a Tool: The paper advocates for the use of thought experiments as a cost-effective way to probe the limits of generalization and identify potential deficiencies in data collection and system design, especially for safety-critical applications.

Key Results

The thought experiment, applied to the LEGO® brick binning task, yielded several key findings summarized in Table 2:

  • Color Bias: Leaving out primary colors (red, blue, yellow) resulted in a significant generalization error (G_c < 0.166685), far exceeding the acceptable threshold. However, leaving out non-primary colors (green, black, white) did not significantly impact error (G_c < 0.000018).
  • Size Bias: Omitting a single stud size led to a noticeable error increase (G_z < 0.018602).
  • 3D Pose Bias: Leaving out one stable 3D pose resulted in a substantial error (G_p < 0.120018), indicating that pose is a critical dimension.
  • Shape Bias: Excluding an entire category of shapes (slopes) also led to a significant error (G_s < 0.006488), though the authors note this estimate is optimistic.
  • Orientation Bias: Omitting one orientation in the imaging plane did not significantly increase error (G_o < 0.000018), suggesting that this dimension might be more robustly handled by standard data augmentation techniques.

In essence, for the LEGO® domain, biases in color (primary), size, 3D pose, and shape led to generalization errors orders of magnitude greater than the acceptable manufacturing error.

Limitations and Future Work

  • Assumptions: The paper relies on several generous assumptions, such as perfect learning with zero training error and the network being able to handle all required classes. These assumptions are made to isolate the effect of bias.
  • LEGO® Domain Specificity: While the LEGO® domain is well-defined, its simplicity might not fully capture the complexities of real-world natural images or more abstract domains.
  • Combinatorial Biases: The paper primarily focuses on single-dimension biases. The authors acknowledge that combinations of missing attributes would likely lead to even greater errors.
  • Future Work:
    • Exploring other types of bias beyond selection bias.
  • Investigating the impact of combinations of biases.
  • Developing more principled methods for analyzing and addressing data sampling deficiencies.
  • Unifying classical computer vision techniques with modern machine learning to better handle variability in object pose, viewpoint, and lighting.
  • Further research into the role of different color spaces and their impact on generalization.
  • Developing a "Bias Breakdown" framework to systematically analyze and document biases in training data.

Summary updated

Abstract

<title>Abstract</title> Learned systems in the domain of visual recognition and cognition impress in part because even though they are trained with datasets many orders of magnitude smaller than the full population of possible images, they exhibit sufficient generalization to be applicable to new and previously unseen data. Since training data sets typically represent such a small sampling of any domain, the possibility of bias in their composition is very real. But what are the limits of generalization given such bias, and up to what point might it be sufficient for a real problem task?<italic> </italic>Although many have examined issues regarding generalization from several perspectives, this question may require examining the data itself. Here, we focus on the characteristics of the training data that may play a role. Other disciplines have grappled with these problems also, most interestingly epidemiology, where experimental bias is a critical concern. The range and nature of data biases seen clinically are really quite relatable to learned vision systems. One obvious way to deal with bias is to ensure a large enough training set, but this might be infeasible for many domains. Another approach might be to perform a statistical analysis of the actual training set, to determine if all aspects of the domain are fairly captured. This too is difficult, in part because the full set of important variables might not be known, or perhaps not even knowable. Here, we try a different, simpler, approach in the tradition of the <italic>Thought Experiment,</italic> whose most famous instance is perhaps Schrödinger's Cat, to address part of these problems. There are many types of bias as will be seen, but we focus only on one, selection bias. The point of the thought experiment is not to demonstrate problems with all learned systems. Rather, this might be a simple theoretical tool to probe into bias during data collection to highlight deficiencies that might then deserve extra attention either in data collection or system development.
BibTeX
@inproceedings{tsotsos2021probing,
  author = {John K. Tsotsos and Jun Luo},
  title = {{Probing the Effect of Selection Bias on Generalization: A Thought Experiment}},
  booktitle = {Research},
  year = {2021},
  doi = {10.21203/rs.3.rs-1117982/v1},
}

Loading related content...

Figure