Abstract: Learned networks in the domain of visual recognition and cognition impress in part because even though they are trained with datasets many orders of magnitude smaller than the full population of possible images, they exhibit sufficient generalization to be applicable to new and previously unseen data. Although many have examined issues regarding generalization from several perspectives, we wondered: If a network is trained with a biased dataset that misses particular samples corresponding to some defining domain attribute, can it generalize to the full domain from which that training dataset was extracted? It is certainly true that in vision, no current training set fully captures all visual information and this may lead to Selection Bias. Here, we try a novel approach in the tradition of the Thought Experiment. We run this thought experiment on a real domain of visual objects that we can fully characterize and look at specific gaps in training data and their impact on performance requirements. Our thought experiment points to three conclusions: first, that generalization behavior is dependent on how sufficiently the particular dimensions of the domain are represented during training; second, that the utility of any generalization is completely dependent on the acceptable system error; and third, that specific visual features of objects, such as pose orientations out of the imaging plane or colours, may not be recoverable if not represented sufficiently in a training set. Any currently observed generalization in modern deep learning networks may be more the result of coincidental alignments and whose utility needs to be confirmed with respect to a system's performance specification.
Introduction
In the field of machine learning, "generalization"—the ability of a model to perform well on unseen data—is often treated as a given. However, training sets are infinitesimally small compared to the "universe" of all possible images. This discrepancy leads to selection bias, where certain attributes of a domain are underrepresented or entirely missing. This paper addresses a critical question: If a network is trained on a biased dataset that completely misses a specific domain attribute (e.g., it never sees a yellow object), can it truly generalize to the full domain while maintaining the high accuracy required for real-world engineering?
While researchers often focus on improving generalization through techniques like data augmentation, few have explored the theoretical limits of generalization when faced with "data holes" in high-stakes environments like autonomous driving or industrial automation. This study uses a rigorous thought experiment to challenge the assumption that deep learning models can reliably generalize across major gaps in training data.
Methodology: The Thought Experiment
The study employs a "Thought Experiment" using two specific logical tools to probe the limits of neural network performance:
- Backcasting: Rather than predicting future performance, the analysis starts with a specific "future" state—a perfectly functioning industrial sorting system—and works backward to identify the logical mechanisms and data requirements needed to achieve that state.
- Reductio ad Absurdum: The authors assume the prevailing belief—that networks can generalize across any bias—is true, and then demonstrate how this assumption leads to logical contradictions when applied to physics and geometry.
The hypothetical testbed is an industrial conveyor belt designed to sort LEGO® bricks with an error rate lower than $1.8 \times 10^{-5}$, matching the actual manufacturing standards of the LEGO Group.
Characterizing the LEGO® Bricks World
The LEGO® Bricks World serves as the theoretical dataset because the domain is fully "knowable" and documented. The scale of this universe is vast: it includes 520 unique brick types, 141 colors, 72 possible orientations on a belt, and an average of 3 stable 3D poses per brick. This results in approximately 15.8 million possible unique images for a single instance of each part.
As shown in the breakdown above, the 520 unique brick types are distributed across various functional categories. By comparing this scale to benchmarks like ImageNet, the authors illustrate how even "massive" datasets are merely tiny samplings of reality.
Results: The Bias Breakdown
The authors analyzed five cases of selection bias. In nearly every scenario, the "generalization error" far exceeded the required engineering specification ($G < 0.000018$):
| Training Set Bias | Resulting Generalization Error | Verdict |
|---|---|---|
| Missing One Primary Color | ~16.6% | Fail (Primary colors cannot be "guessed") |
| Missing One Brick Size | ~1.8% | Fail (Unique shapes cannot be composed) |
| Missing Orientation | No increase | Pass (Solved by rotation augmentation) |
| Missing 3D Pose | ~12.0% | Fail (Aspect graphs are topologically incomplete) |
| Missing Shape (Slopes) | ~0.6% (Optimistic) | Fail (NP-Complete complexity issues) |
Key Insights
- Generalization vs. Engineering Specs: A model that is "90% accurate" may be a research success but is an engineering failure for safety-critical systems. The utility of generalization is entirely dependent on the acceptable system error.
- The "Primary Attribute" Wall: Certain features are physically or mathematically unrecoverable. If a network never sees "Yellow," it cannot recognize yellow bricks because the necessary weights were never triggered during training.
- The Hidden Cost of Data Augmentation: Common techniques like rotating images have secondary effects. Rotating an image changes the distribution of light source directions, which can confuse agents relying on shadows for depth perception.
Conclusion
This thought experiment suggests that any currently observed generalization in deep learning may be the result of coincidental alignments rather than robust learning. For safety-critical domains, current empirical methodologies are insufficient. The authors propose a new standard for AI research: a "Bias Breakdown" table. Instead of just reporting aggregate accuracy, researchers should explicitly document which dimensions of the domain were not sampled, allowing for a more honest assessment of a model's limitations in real-world applications.
Summary updated