When the state of the art is ahead of the state of understanding: Unintuitive properties of deep neural networks

Joan SerràMètode 2018 · DOI

Abstract

Deep learning has achieved unprecedented success across academia and industry, driven by the availability of massive datasets and significant computing power. However, this empirical progress has largely outpaced our theoretical understanding of how these models function. Deep neural networks (DNNs) exhibit several unintuitive properties—such as the ability to memorize random noise while maintaining generalization—that challenge classical statistical learning theory. This paper highlights these paradoxes and emphasizes the urgent need for a more rigorous theoretical framework to bridge the "understanding gap."

Introduction: The "Wild West" of Deep Learning

In most scientific disciplines, theory and practice advance in tandem. Deep learning, however, has entered a phase where practitioners achieve record-breaking results using empirical "tricks of the trade"—such as specific weight initializations or ReLU activations—that lack a unified mathematical foundation. While DNNs are loosely inspired by biological brains, their behavior often contradicts classical wisdom regarding optimization and generalization.

A standard deep neural network architecture consisting of input, hidden, and output layers.

As illustrated in Figure 1, the basic structure of a DNN involves layers of interconnected nodes. Despite this structural simplicity, the interaction between millions of parameters creates a complex system whose global behavior remains difficult to predict or explain from first principles.

The Understanding Gap

The current state of research mirrors Stephen Wolfram’s "new kind of science" paradigm, where optimal designs are discovered through combinatorial search and empirical brute force rather than derived from mathematical proofs. This has led to a series of "aha moments" where DNN behavior defies human intuition.

The "Dumb Error" Paradox and Adversarial Examples

One of the most striking contradictions is that DNNs can reach superhuman accuracy on complex tasks yet remain vulnerable to "adversarial examples." These are imperceptible perturbations to an input that lead to high-confidence misclassifications. This highlights a fundamental difference between how machines and humans perceive features.

The classic "chihuahua vs. muffin" challenge illustrating the visual ambiguities that can complicate classification.

As seen in Figure 4, even for humans, certain visual patterns (like the "chihuahua vs. muffin" problem) require careful discernment. For a DNN, these similarities can be exploited by adversarial noise, causing a network to identify random pixels as a recognizable object with over 99.99% confidence.

Memorization vs. Generalization

Classical learning theory suggests that a model powerful enough to memorize random noise should fail to generalize to new, unseen data. DNNs break this rule. Research by Zhang et al. (2017) demonstrated that networks can achieve 100% training accuracy on datasets with completely random labels, proving they have the capacity for total memorization, yet they still perform exceptionally well on real-world data.

The Compression Paradox

There is a strange inefficiency in how we train these models. We often require massive, over-parameterized networks to learn a task effectively. However, once training is complete, these networks can often be pruned or "distilled" by up to 100x—removing the vast majority of their weights—without any significant loss in performance. This raises a critical question: why is the "fat" network necessary for learning if the "thin" network is sufficient for execution?

Conclusion and Future Directions

The primary challenge facing the field is the lack of a formal theory to replace the current reliance on empirical trial-and-error. We currently lack mathematically-motivated schemes for optimal initialization or understanding the sensitivity of networks to the order of training samples.

Moving forward, the field is ripe for a paradigm shift. While empirical exploration is currently the dominant mode of progress, even minor theoretical advancements could lead to a more holistic understanding. The ultimate goal is to move beyond a collection of "tricks" and develop a theory that explains why these unintuitive properties exist and how to harness them more reliably for the next generation of artificial intelligence.

Summary updated

Abstract

Deep learning is an undeniably hot topic, not only within both academia and industry, but also among society and the media. The reasons for the advent of its popularity are manifold: unprecedented availability of data and computing power, some innovative methodologies, minor but significant technical tricks, etc. However, interestingly, the current success and practice of deep learning seems to be uncorrelated with its theoretical, more formal understanding. And with that, deep learning’s state-of-the-art presents a number of unintuitive properties or situations. In this note, I highlight some of these unintuitive properties, trying to show relevant recent work, and expose the need to get insight into them, either by formal or more empirical means.
BibTeX
@inproceedings{serra2018when,
  author = {Joan Serrà},
  title = {{When the state of the art is ahead of the state of understanding: Unintuitive properties of deep neural networks}},
  booktitle = {Mètode},
  year = {2018},
  doi = {10.7203/metode.9.11035},
}
Figure