Boosting Federated Domain Generalization: Understanding the Role of Advanced Pretrained Architectures

Avi Deb Raha, Kitae Kim, Apurba Adhikary, Mrityunjoy Gain, Yu Qiao, Zhu Han, Choong Seon HongIEEE Internet of Things Journal 2025 · DOI

TL;DR

This paper investigates how advanced pre-trained neural network architectures (like Vision Transformers and ConvNeXt) and larger pre-training datasets can significantly boost Federated Domain Generalization (FDG). The findings show that these modern approaches establish new benchmarks, outperforming traditional methods and offering more efficient solutions for privacy-preserving, cross-domain learning.

What Problem Does This Paper Address?

Federated Learning (FL) allows collaborative model training without sharing raw data, preserving privacy. However, a major challenge is domain generalization (DG): ensuring models perform well on data from new, unseen domains, which is crucial in real-world scenarios where data distributions vary. Existing Federated Domain Generalization (FDG) methods often rely on older architectures like ResNet pre-trained on ImageNet-1K. This can limit adaptability due to architectural constraints and the limited scope of the pre-training data. This paper aims to bridge this gap by exploring how cutting-edge architectures and larger, more diverse pre-training datasets can enhance FDG performance.

Methods

The core of this research involves evaluating various advanced neural network architectures, including Vision Transformers (ViT), Swin Transformers, and ConvNeXt, within the FDG framework. These architectures are chosen for their ability to capture global context and long-range dependencies, which are beneficial for generalization. The study systematically compares:

  • Different architectures: Exploring variants of ConvNeXt, ViT, and Swin Transformers, as well as traditional ResNet models.
  • Pre-training datasets: Utilizing a range of datasets from ImageNet-1K to larger ones like ImageNet-21K, ImageNet-22K, and JFT-300M.
  • Pre-training strategies: Comparing self-supervised learning (SSL) techniques (like MoCov3, DINO, SwAV, and BEiT) against traditional supervised pre-training.

The FL process involves clients training local models on their private data, with the central server aggregating these updates to form a global model. The study uses a leave-one-domain-out evaluation protocol for rigorous testing.

Datasets

The paper uses the following datasets:

  • Pre-training Datasets: ImageNet-1K, ImageNet-12K, ImageNet-21K, ImageNet-22K, and JFT-300M.
  • Downstream Datasets (for FDG evaluation):
    • Office-Home: Contains four domains (Product, Art, Clipart, Real-world) with smaller domain discrepancies.
  • PACS: Consists of four domains (Photo, Art painting, Cartoon, Sketch) with more distinct domain variations.

Key Insights and Ideas

  • Advanced Architectures are Key: Modern architectures like ConvNeXt and Swin Transformers significantly outperform traditional ResNet models in FDG, even when using smaller variants.
  • Larger Pre-training Datasets Matter: Pre-training on more extensive datasets (ImageNet-21K, ImageNet-22K, JFT-300M) consistently leads to better FDG performance, especially for complex datasets with subtle domain differences.
  • Self-Supervised Learning (SSL) Nuances: SSL methods, particularly BEiT, show strong generalization, especially on datasets with distinct domains. However, on datasets with subtle domain differences, supervised pre-training can sometimes perform better.
  • Model Size vs. Performance: Surprisingly, some smaller, advanced models (like ConvNeXt-N) can outperform larger ResNet models, highlighting the importance of architectural design and pre-training strategy over sheer parameter count.
  • ConvNeXt Dominance: ConvNeXt models, particularly when pre-trained on larger datasets like ImageNet-22K, consistently achieve state-of-the-art results on both Office-Home and PACS datasets.

Key Results

  • New Benchmarks: The study establishes new benchmarks for FDG, achieving average accuracies of 84.46% on Office-Home and 92.55% on PACS using advanced architectures pre-trained on larger datasets.
  • ConvNeXt Superiority: ConvNeXt-B pre-trained on ImageNet-22K achieved the highest overall accuracy on PACS (92.55%) and Office-Home (84.46%).
  • SSL Performance: BEiT demonstrated strong performance, achieving an average accuracy of 78.12% on Office-Home and 88.04% on PACS, outperforming supervised counterparts in many cases.
  • Architectural Depth: For ResNet, increasing depth improved accuracy, but advanced architectures like ConvNeXt showed more consistent gains with depth.
  • Parameter Efficiency: Models like ConvNeXt-A and ConvNeXt-P, with fewer parameters, outperformed ResNet-18, demonstrating architectural efficiency.

Limitations and Future Work

  • Dataset Availability: The study notes that pre-trained models for some variants (e.g., ConvNeXt-A, ConvNeXt-P) on larger datasets were not publicly available, limiting the scope of comparison.
  • SSL vs. Supervised: The paper highlights that the effectiveness of SSL versus supervised pre-training depends on the dataset's domain characteristics (subtle vs. distinct differences). Future work could explore hybrid approaches or more tailored SSL methods.
  • Resource Constraints: While advanced architectures offer better performance, their computational demands in federated settings (especially on edge devices) remain a consideration. Future research could focus on optimizing these models for resource-constrained environments.

Summary updated

BibTeX
@article{raha2025boosting,
  author = {Avi Deb Raha and Kitae Kim and Apurba Adhikary and Mrityunjoy Gain and Yu Qiao and Zhu Han and Choong Seon Hong},
  title = {{Boosting Federated Domain Generalization: Understanding the Role of Advanced Pretrained Architectures}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2025.3579372},
}
Figure