Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection

Taeheon Lim, Joohyung Lee, Kyungjae Lee, Jungchan ChoarXiv 2025 · DOI · arXiv

Introduction

Federated Learning (FL) has traditionally focused on overcoming class-level heterogeneity—where different clients possess different categories of data. However, real-world deployments involve a diverse array of hardware, from high-end smartphones to legacy CCTV cameras, which capture data at vastly different resolutions. This paper introduces Resolution-Adaptive Federated Learning (RAF) to address "resolution-drift," a phenomenon where global models fail because they are aggregated from clients with mismatched image scales.

Resolution heterogeneity represents a distinct axis of data distribution challenges in Federated Learning, separate from traditional class heterogeneity.

The Problem: Resolution-Drift

In high-resolution regression tasks like Human Pose Estimation (HPE), spatial detail is paramount. The authors identify resolution-drift as a critical failure mode: when a global model is aggregated from clients using different resolutions, it often overfits to specific scales and loses generalization. Standard FL algorithms like FedAvg or FedProx struggle in these scenarios because they are not designed to handle the "feature representation mismatch" caused by varying pixel densities across the network.

The RAF Framework

The proposed RAF framework tackles this problem through two primary innovations:

  1. Architectural Flexibility: Standard Vision Transformers (ViTs) often use Absolute Positional Embeddings (APE) that are hard-coded to specific input sizes. RAF replaces these with convolution-based positional embeddings (Global and Local), allowing the model to process images of any size without needing to resize or interpolate the input first.
  2. Multi-Resolution Knowledge Distillation (MRKD): This is the core mechanism of RAF. During local training, a client doesn't just train on its native resolution; it creates downsampled versions of its images and uses the high-resolution output as a "teacher" to guide the lower-resolution "student" outputs. By minimizing the difference (MSE) between these heatmaps, the model learns to extract consistent spatial features regardless of the input scale.
The RAF framework architecture, showing how clients with diverse hardware and resolutions contribute to a unified global model through resolution-adaptive updates.

Key Insights and Results

The study, conducted using the MPII Human Pose dataset, demonstrates that resolution is a critical "Non-IID" axis. Key findings include:

  • Accuracy Boosts: In mixed-resolution scenarios, RAF improved accuracy by 5.4% at the lowest resolution (128x96) and 3.9% at high unseen resolutions (512x384) compared to standard FedAvg.
  • Mitigating Drift: When trained exclusively on high-res data and tested on low-res inputs, RAF yielded a massive 27.6% improvement over the baseline, effectively "rescuing" the model's performance on poor-quality data.
  • Regularization Effect: MRKD acts as a powerful regularizer. Even if a client only has high-resolution data, training with the multi-resolution loss prevents the model from becoming "brittle" and helps it perform better on low-resolution images it has never seen.
  • Feature Organization: t-SNE visualizations (not pictured) confirmed that while standard models collapse features from different resolutions into messy clusters, RAF-trained models create organized, distinct clusters for each scale, indicating the network has successfully learned to handle varying levels of detail.

Conclusion

RAF demonstrates that preserving spatial fidelity is essential for regression tasks in federated environments. By making models resolution-agnostic, RAF enables robust performance across the heterogeneous landscape of modern edge devices, from high-definition mobile photography to low-resolution security monitoring.

Summary updated

BibTeX
@misc{lim2025mitigating,
  author = {Taeheon Lim and Joohyung Lee and Kyungjae Lee and Jungchan Cho},
  title = {{Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection}},
  year = {2025},
  eprint = {2507.23461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.23461},
}
Figure