Abstract
This paper presents a study on an automated system for image classification, which is based on the fusion of various deep learning methods. The study explores how to create an ensemble of different Convolutional Neural Network (CNN) models and transformer topologies that are fine-tuned on several datasets to leverage their diversity. The research question addressed in this work is whether different optimization algorithms can help in developing robust and efficient machine learning systems to be used in different domains for classification purposes. To do that, we introduce novel Adam variants. We employed these new approaches, coupled with several CNN topologies, for building an ensemble of classifiers that outperforms both other Adam-based methods and stochastic gradient descent. Additionally, the study combines the ensemble of CNNs with an ensemble of transformers based on different topologies, such as Deit, Vit, Swin, and Coat. To the best of our knowledge, this is the first work in which an in-depth study of a set of transformers and convolutional neural networks in a large set of small/medium-sized images is carried out. The experiments performed on several datasets demonstrate that the combination of such different models results in a substantial performance improvement in all tested problems. All resources are available at https://github.com/LorisNanni .
Introduction
In the field of deep learning, the pursuit of state-of-the-art performance often centers on finding a single superior architecture. However, individual models—whether they are Convolutional Neural Networks (CNNs) or the newer Vision Transformers—possess distinct "blind spots." CNNs typically focus on local textures, while Transformers excel at capturing global shapes and long-range dependencies. This research demonstrates that the key to robust image classification is not just a better architecture, but a diverse ensemble powered by novel optimization strategies.
Methodology
The core of the proposed system is a "fusion of fusions," combining multiple architectural families with diverse optimization paths. The ensemble consists of:
- CNN Topologies: Specifically ResNet50 and EfficientNetB0.
- Transformer Topologies: Including DeiT, ViT, Swin, and CoAtNet.
To maximize the diversity of the ensemble, the researchers introduced three new Adam-based optimization variants used during the training phase:
- Hyperbolic (Hyp): Uses a hyperbolic weighting function for the learning rate to handle gradient changes more aggressively, leading to faster convergence.
- MinD: A "minimum rule" optimizer that combines the strengths of DGrad and Exp optimizers to smooth the optimization path and avoid local minima.
- Angular Injection (AI): This variant analyzes the direction (angle) of consecutive gradients, injecting curvature information into the momentum to navigate complex loss landscapes.
The final prediction is generated using a weighted sum rule, effectively combining the outputs of all diverse models to mitigate individual errors.
Experimental Setup and Datasets
The approach was evaluated across nine datasets spanning various domains to ensure generalizability:
- Marine Biology: WHOI, ZooScan, Kaggle (plankton), and ZooLake.
- Medical Imaging: Breast Grading Carcinoma (BG) and Laryngeal patches (LAR).
- Pests & Viruses: Agricultural pests (Deng), virus types (VIR), and virus particles (TEM).
Key Results and Insights
The experimental results consistently validated the proposed approach:
- State-of-the-Art Performance: The ensemble achieved new state-of-the-art (SOTA) results on several benchmarks, including the WHOI, LAR, and Deng datasets.
- Statistical Significance: Ensembles utilizing the new Adam variants (Hyp+MinD+AI) significantly outperformed those using standard Adam or Stochastic Gradient Descent (SGD) (p < 0.0001).
- Architectural Complementarity: Combining CNNs and Transformers outperformed ensembles consisting of only one type (p < 0.01), confirming that "texture experts" and "shape experts" work better together.
- Optimization as a Diversity Tool: A primary insight is that changing the optimizer is as effective for ensemble diversity as changing the architecture itself.
Conclusion and Future Work
This study highlights that diversity is a fundamental strength in deep learning. By fusing CNNs and Transformers through novel optimization techniques, we can create highly accurate and robust classification systems that require minimal dataset-specific tuning.
Future research will focus on developing optimizers specifically tailored for Transformer architectures and exploring knowledge distillation techniques to compress these large ensembles into smaller, more efficient models for real-time applications.
All resources are available at https://github.com/LorisNanni.
Summary updated