De la réduction de la complexité des modèles à la sélection des caractéristiques dans l'apprentissage profond : une histoire de régularisation

Enzo TartaglioneSPIRE 2024 · OpenAlex

Abstract: In the last decade, the scientific community has witnessed the blooming and the massive exploitation of deep neural networks (DNNs). This is fueled by multiple factors: the inherent flexibility offered by DNNs to learn input-output functions from big data and the computation scaling-up in computational capability offered by computing devices, which grasped the attention of a growing research community that progressively further improves their performance. Such a tsunamic trend finds its (momentary) apex in foundation models: DNNs trained on a dementially large quantity of data, are able to extract rich features that adapt to a broad range of downstream tasks. Siding this enthusiasm in achieving generality for these models, multiple problems were encountered, especially when dealing with the model’s usability in resource-constrained environments, or when learning from biased data. In this work, an analysis of these two aspects will be conducted, also showing how the two are intrinsically linked. This work summarizes research conducted after the PhD, divided mainly into two parts: model compression and efficiency (with a special emphasis on deep neural network pruning) and model debiasing (with openings also to privacy for DNNs). In all the presented approaches, the recurrent underlying theme, either in an implicit or explicit fashion, will be the design of regularization for DNNs.

1. Introduction: The Dual Challenge of Modern AI

Modern Deep Learning is currently navigating a "tsunami" of complexity. As foundation models grow increasingly massive, they become difficult to deploy on edge devices like smartphones or IoT sensors due to prohibitive memory and energy demands. Simultaneously, these models are prone to learning "shortcuts"—spurious correlations in training data that lead to algorithmic bias. For instance, a medical imaging model might learn to identify a disease based on the specific hospital equipment visible in the background rather than the actual pathology.

This paper argues that these two challenges—complexity reduction (making models smaller) and feature selection (making models focus on relevant data)—are two sides of the same coin. The core problem is distinguishing "relevant information" from "noise," and the solution lies in the strategic application of regularization.

2. Methods for Model Efficiency and Compression

To achieve "frugal AI," several novel regularization-based approaches are introduced:

  • Sensitivity-based Pruning (SeReNe & LOBSTER): Rather than simple magnitude pruning (removing small weights), these methods utilize gradients to identify parameters with negligible impact on the final output. SeReNe focuses on "structured sparsity," removing entire neurons or filters to ensure physical speedups on standard hardware.
  • Layer Folding (NEPENTHE): This technique uses entropy measures to identify layers that have become functionally linear. These layers can be mathematically merged or "folded" into adjacent layers, reducing the total depth of the network without accuracy loss.
  • Neurons at Equilibrium: To optimize the training phase, this method identifies neurons whose functions have stabilized. By "pinning" these neurons and skipping their backpropagation, training FLOPs can be reduced by up to 60% in Transformer architectures.

3. Feature Selection: Debiasing and Privacy

Ensuring model fairness and trustworthiness requires filtering out biased or private information:

  • Debiasing via EnD & IRENE: The EnD (Entangling and Disentangling) regularizer forces the model to separate target features from bias features. IRENE acts as an information filter at the network's bottleneck, explicitly removing biased or private information using an auxiliary classifier.
  • Unsupervised Bias Mining: A novel method using Voronoi boundaries in the latent space detects when a model begins to "fit the bias" during training. This allows for correction even when explicit bias labels are unavailable.

4. Theoretical Insights and the "Sparse Double Descent"

Beyond practical methods, this work explores the theoretical landscape of pruning:

  • The Sparse Double Descent: In noisy data environments, pruning can cause a counter-intuitive phenomenon where error initially increases before entering a "sweet phase" of improved performance. Entropy-based metrics are provided to help practitioners navigate this transition.
  • Structured vs. Unstructured Sparsity: While unstructured pruning achieves higher theoretical compression, structured pruning is significantly more effective for real-world latency improvements on general-purpose CPUs and GPUs.
  • The Lottery Ticket Landscape: Pruning alters the "loss landscape" of a model, making it progressively flatter and harder to optimize, which explains why iterative pruning during training is more effective than "zero-shot" pruning at initialization.

5. Key Results and Empirical Validation

The research validates these methods across benchmarks including CIFAR-10/100, ImageNet, and medical Chest X-Ray datasets:

  • Efficiency: SeReNe achieved significant inference speedups on embedded devices (Raspberry Pi) compared to unpruned models.
  • Fairness: On the CelebA dataset, the unsupervised bias mining approach achieved a test accuracy of 90.2% on unbiased samples, significantly outperforming vanilla models (79.0%) and remaining competitive with supervised methods.

6. Conclusion and Future Directions

The recurrent theme of this research is that regularization is the key to balancing performance, efficiency, and fairness. Future work will investigate "layer collapse" in massive multimodal models (like CLIP or LLMs) and move toward "bias-agnostic" training, where models can autonomously sense and ignore spurious correlations in real-time.

Summary updated

Abstract

In the last decade, the scientific community has witnessed the blooming and the massive exploitation of deep neural networks (DNNs). This is fueled by multiple factors: the inherent flexibility offered by DNNs to learn input-output functions from big data and the computation scaling-up in computational capability offered by computing devices, which grasped the attention of a growing research community that progressively further improves their performance. Such a tsunamic trend finds its (momentary) apex in foundation models: DNNs trained on a dementially large quantity of data, are able to extract rich features that adapt to a broad range of downstream tasks.Siding this enthusiasm in achieving generality for these models, multiple problems were encountered, especially when dealing with the model’s usability in resource-constrained environments, or when learning from biased data. In this work, an analysis of these two aspects will be conducted, also showing how the two are intrinsically linked. This work summarizes my research conducted after the PhD, divided mainly into two parts:- In the first part, model compression and efficiency will be treated, with a special emphasis on deep neural network pruning;- In the second part, model debiasing is treated, with openings also to privacy for DNN (intended as feature hiding);All along these, new research trends currently under study are suggested: the employment of adapters for efficient fine-tuning of a large pre-trained model, training with a selection of a subset of neurons to update, on-device learning with improved selection of a DNN to fine-tune, DNN depth reduction, and more.In all the presented approaches, the recurrent underlying theme, either in an implicit or explicit fashion, will be the design of regularization for DNNs.
BibTeX
@inproceedings{tartaglione2024de,
  author = {Enzo Tartaglione},
  title = {{De la réduction de la complexité des modèles à la sélection des caractéristiques dans l'apprentissage profond : une histoire de régularisation}},
  booktitle = {SPIRE},
  year = {2024},
}
Figure