Abstract
Deep Neural Networks (DNNs) often achieve high accuracy by exploiting "simplicity bias"—relying on easy, irrelevant shortcuts rather than learning core semantic concepts. This paper presents a diagnostic framework that reconstructs training samples directly from a model's weights to assess what features have been prioritized. By analyzing these recovered images, we can determine if a model has truly captured the intended concepts or merely memorized superficial triggers.
Introduction: The Challenge of Simplicity Bias
Despite their success, DNNs remain "black boxes" prone to latching onto the simplest features to distinguish classes, such as background colors or watermarks. Standard generalization tests on separate validation sets can identify performance failures but rarely explain the underlying cause. Furthermore, curating comprehensive test sets for every potential edge case is prohibitively expensive. This research introduces a cost-effective diagnostic tool to reveal what a model has "memorized" into its parameters without requiring external validation data.
Methodology: Knowledge Recovery and Bias Assessment
The proposed framework utilizes a Knowledge Recovery module and a Score Function to evaluate model integrity.
Knowledge Recovery
Drawing on privacy research, the authors employ a reconstruction scheme that treats a trained model's weights as a distilled representation of the training data. For homogeneous ReLU networks, gradient flow converges toward a "max-margin" solution. By minimizing a specific reconstruction loss function, the system synthesizes images that represent the features the model's weights have prioritized.
Bias Injection and Evaluation
To validate the method, datasets were intentionally "poisoned" with triggers of varying complexity:
- Level One (Easy): A fixed square in the top-left corner.
- Level Two (Medium): Triggers in three random positions per class.
- Level Three (Hard): Triggers in five random positions per class.
The Structural Similarity Index (SSIM) is used to compare reconstructed samples against original training data. High-quality reconstruction of the "main concept" (e.g., the object) indicates an unbiased model, whereas clear reconstruction of only the trigger or background signals a biased model.
Experimental Results
The method was evaluated on MNIST (binary odd/even) and CIFAR-10 (binary vehicle/animal).
Bias Visualization
In models trained with "Level One" bias, reconstructed images failed to depict the actual objects, confirming the model ignored them in favor of the simple trigger. As shown in the recovered samples, the model focuses almost entirely on the injected square, leaving the rest of the image as a non-descript blur.
The "Harder is Better" Paradox
A key finding is that complex biases (Level Three) actually improved the learning of primary concepts. Because the shortcut was harder to find, the model was forced to learn the actual features of the object. Conversely, simple biases caused the model to stop learning main features almost immediately.
Metric Analysis
CIFAR-10 reconstructions yielded significantly higher SSIM scores (0.5182) than MNIST (0.1862), suggesting that more complex datasets force models to learn more discriminative, and thus more reconstructible, features. Additionally, contrast-based biases were found to be less "distracting" than geometric triggers, with models still managing to reconstruct main objects well (SSIM 0.5774).
Discussion and Future Work
While promising, the current reconstruction algorithm is primarily effective for fully connected, homogeneous ReLU networks and faces challenges scaling to deep CNNs or Transformers. Future work aims to remove the dependency on "margin" data by using non-reference image quality metrics and to adapt the framework for high-resolution, large-scale datasets. Currently, human visual inspection remains a critical component for simple datasets like MNIST where automated metrics can be misleading.
Summary updated