Identifying and exploiting structures for reliable deep learning

Amartya Sanyal

Modern deep neural networks (DNNs) exhibit a fundamental irony: as they grow in capacity and accuracy, they often become less reliable. High-capacity models frequently memorize noise, succumb to imperceptible adversarial attacks, become overconfident in their errors, and struggle to protect user privacy during inference. This research, a doctoral thesis from the University of Oxford, argues that these "side effects" can be mitigated by identifying and exploiting specific mathematical structures within data and model architectures.

The Reliability Gap in Deep Learning

While deep learning has achieved state-of-the-art performance in vision and language tasks, reliability remains a significant hurdle for deployment in high-stakes environments like healthcare or autonomous driving. Four critical failure modes are addressed:

  1. Generalization Gap: Models often "memorize" training data rather than learning underlying logic, leading to failure on unseen data.
  2. Adversarial Vulnerability: Tiny, imperceptible perturbations to inputs can flip a model’s prediction with high confidence.
  3. Miscalibration: Modern DNNs are frequently "confidently wrong," where predicted probabilities do not reflect actual accuracy.
  4. Privacy Costs: Running models on encrypted private data is currently too computationally expensive for real-time applications.

Proposed Structural Interventions

To transform deep learning from a "black box" into a reliable tool, four structural interventions are introduced:

  • Stable Rank Normalization (SRN): Rather than standard weight regularization, SRN controls the "stable rank" of weight matrices. This limits the model's ability to use excess capacity for memorizing random noise, forcing it to focus on more informative features.
  • Low-Rank (LR) Representations: By introducing a "virtual layer" that constrains the model to a low-dimensional subspace, the model is forced to rely on principal features. This reduces the available "noisy" directions that an adversary can exploit.
  • Focal Loss for Calibration: While typically used for imbalanced data, this work proves that Focal Loss acts as a regularizer that prevents weights from growing excessively large. This keeps the model "humble," ensuring confidence scores are better calibrated to actual performance.
  • Encrypted Prediction (EPAAS): To accelerate privacy-preserving AI, the research utilizes Binary Neural Networks (BNNs)—where weights are restricted to -1 or +1—paired with Fully Homomorphic Encryption (FHE). This allows encrypted data processing via simple logical gates rather than complex arithmetic.

Experimental Validation

These methods were validated across diverse benchmarks, including standard image classification (MNIST, CIFAR-10/100, SVHN), large-scale vision (Tiny-ImageNet), generative modeling (CelebA), and text analysis (SST). Privacy-sensitive applications were tested using healthcare datasets (Cancer and Diabetes) and facial recognition (LFW-a).

Key Results

  • Generalization: SRN improved ResNet-110 accuracy on CIFAR-100 from 71.5% to 73.1% while suppressing the memorization of random labels.
  • Robustness: Models utilizing the Low-Rank layer demonstrated up to a 10x increase in accuracy under white-box adversarial attacks compared to standard models, without sacrificing clean-data performance.
  • Calibration: Focal Loss reduced the Expected Calibration Error (ECE) on CIFAR-100 from 17.5% to approximately 5%, outperforming specialized techniques like Temperature Scaling.
  • Privacy Efficiency: The BNN-based encrypted prediction framework achieved a 29x speedup on MNIST compared to previous state-of-the-art encrypted methods.

Insights and Future Directions

A critical theoretical insight from this work is the Robustness-Memorization Link: the proof that memorizing even a small amount of label noise makes a model fundamentally impossible to defend against adversarial attacks. Furthermore, the research suggests that standard training (SGD) often finds decision boundaries that are too simple and pass too close to data points; adding complexity through richer, low-rank representations actually creates a larger safety margin.

While these structural priors significantly enhance reliability, they do not entirely eliminate the trade-off between robustness and accuracy. Future work involves scaling these interventions to larger architectures and exploring the interplay between privacy, robustness, and calibration within a unified framework.

Summary updated

BibTeX
@misc{sanyal2021identifying,
  author = {Amartya Sanyal},
  title = {{Identifying and exploiting structures for reliable deep learning}},
  year = {2021},
  eprint = {2108.07083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.07083},
}
Figure