Abstract
In the era of data-centric AI, the principles governing dataset curation remain largely intuitive rather than theoretical. This paper deconstructs two prevalent myths: that datasets should aim for low "intrinsic dimension" (ID) and that data augmentations must minimize distribution shift from the "natural" data. Through empirical counter-examples using Mixed Sample Data Augmentation (MSDA) techniques like MixUp and FMix, we demonstrate that higher-dimensional representations and "unnatural" shifts can actually enhance model performance. Our findings suggest that the quality and direction of representational bias are more critical than simple compression or distribution preservation.
Introduction
The machine learning community lacks theory-informed guidelines for dataset curation, often relying on heuristics like "keep the data distribution natural" or "aim for simpler, compressed representations." This paper specifically tackles two prominent ideas:
- Intrinsic Dimension (ID): The belief that data lies on a low-dimensional manifold and that minimizing this dimension makes it easier for a model to generalize.
- Distribution Preservation: The idea that data augmentations should stay as close as possible to the original "natural" data distribution (high affinity) to be effective.
Methodology
To test these myths, the authors employ several key techniques:
- Intrinsic Dimension Estimation: They use the TWO-NN algorithm, which estimates the dimension of the data manifold by looking at the distance ratios between a point and its two nearest neighbors in the model's representation space.
- Mixed Sample Data Augmentation (MSDA): They compare standard techniques like MixUp (linear interpolation) and FMix (masking regions in Fourier space).
- RMixUp (Reformulated MixUp): This control variant creates mixed images like MixUp but assigns the label of only one source image. This is designed to force the model to compress its representations (lowering ID) while removing the beneficial regularization of label mixing.
- Inter-dataset Augmentation: They perform "unnatural" mixing by combining images from entirely different datasets (e.g., CIFAR-10 mixed with FashionMNIST) to see if the "naturalness" of the distribution shift actually matters.
The Intrinsic Dimension Myth
The belief that lower intrinsic dimension guarantees better generalization is challenged by the RMixUp experiment. On CIFAR-10, standard MixUp achieved 93.79% accuracy with an ID of 9.14. The RMixUp variant successfully lowered the ID to 7.80 (matching the baseline), but accuracy dropped to 92.40%. This proves that lower ID is not a reliable predictor of better performance; the quality of the features matters more than the degree of manifold compression.
The Distribution Shift Myth
It is often assumed that "affinity"—how much an augmented sample resembles a real one—is key to effective augmentation. However, the authors found that FMix, which creates very "unnatural" looking images, often outperforms MixUp despite having a much larger distribution shift (lower affinity).
The authors argue that the direction of the bias introduced by data manipulation is more important than the magnitude of the shift. Even mixing CIFAR images with FashionMNIST (a massive, "unnatural" shift) did not necessarily destroy performance and sometimes improved it compared to the baseline.
Conclusion
This research highlights that common data-centric heuristics can be misleading. Lower intrinsic dimension does not guarantee better generalization, and "naturalness" is not a prerequisite for effective data augmentation. Future work should move away from simple data-space metrics and focus on how data modifications impact the invariance properties of learned representations. Quantifying and measuring the "direction of bias" in high-dimensional space remains an open research challenge.
Summary updated