A Rigorous Behavior Assessment of CNNs Using a Data-Domain Sampling Regime

Shuning Jiang, Wei‐Lun Chao, Daniel Haehn, Hanspeter Pfister, Jian Chen

Introduction

As machine learning models are increasingly deployed to interpret visual data, understanding the nature of their "perception" becomes paramount. While human graphical perception has been a subject of study for decades—most notably through the work of Cleveland and McGill—the internal logic of Convolutional Neural Networks (CNNs) remains largely opaque. This research addresses a critical gap: how does the distribution of training data in the data-domain influence a model's sensitivity, stability, and expertise?

Methodology: The Sampling Regime

The researchers developed a rigorous sampling framework centered on a ratio estimation task—judging the percentage relationship between two bars in a chart. To evaluate model robustness, they employed five standard bar chart types and two common architectures, VGG19 and ResNet50.

Overview of the training data sampling regime and the neural network pipeline.

The core contribution is the systematic comparison of four distinct sampling strategies across the data domain:

  1. IID (Independent and Identical Distribution): A baseline using simple random sampling.
  2. COV (Coverage): Maximizing the "span" by selecting training samples as distant from each other as possible.
  3. ADV (Adversarial): Selecting samples specifically positioned away from test set values.
  4. OOD (Out-of-Distribution): Concentrating samples in a narrow portion of the domain.
Detailed visualization of the four ratio sampling configurations across the data domain (0.06 to 0.99).

Key Insights

Distance as a Predictor of Error

The study found that "training-test distance"—the average distance from a test sample to its nearest training neighbor—is the single best predictor of CNN error. This relationship held remarkably consistent ($r = 0.97$) across different tasks, including cell and node counting.

The Efficiency of Coverage-Based Sampling

The "Coverage" (COV) strategy emerged as a superior method for training robust models. By ensuring an even spread across the data range, COV models maintained high accuracy even when training data was reduced to 3.75% of the original set. This approach required 50% fewer samples than IID to reach comparable performance, significantly reducing training time by up to 96%.

CNNs vs. Human Biases

Unlike humans, who exhibit specific biases based on chart type (e.g., preferring position over length), CNNs acted as more "pure" mathematical reasoners. They showed little preference between chart types, provided the data range was represented in their training. However, CNNs struggled with "small stimuli"—bars shorter than 15–25 pixels—where standard pooling layers likely discard necessary fine-grained information.

Conclusion

CNNs can achieve superhuman accuracy (up to 4x lower error than humans) in controlled visualization tasks, but their reliability is tethered to the data-domain distribution. The success of COV sampling suggests that we can build more efficient and stable models by focusing on domain coverage rather than just volume. Future work should investigate architectural changes to handle small-scale visual features and expand these insights into more complex visual reasoning tasks.

Summary updated

Abstract

We present a data-domain sampling regime for quantifying CNNs' graphic perception behaviors. This regime lets us evaluate CNNs' ratio estimation ability in bar charts from three perspectives: sensitivity to training-test distribution discrepancies, stability to limited samples, and relative expertise to human observers. After analyzing 16 million trials from 800 CNN models and 6,825 trials from 113 human participants, we arrived at a simple and actionable conclusion: CNNs can outperform humans and their biases simply depend on the training-test distance. We show evidence of this simple, elegant behavior of the machines when they interpret visualization images. osf.io/gfqc3 provides registration, the code for our sampling regime, and experimental results.
BibTeX
@article{jiang2026rigorous,
  author = {Shuning Jiang and Wei‐Lun Chao and Daniel Haehn and Hanspeter Pfister and Jian Chen},
  title = {{A Rigorous Behavior Assessment of CNNs Using a Data-Domain Sampling Regime}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2026},
  doi = {10.1109/tvcg.2025.3633829},
}
Figure