Benchmarking Federated Few-Shot Learning for Video-Based Action Recognition

Nguyen Anh Tu, Nartay Aikyn, Nursultan Makhanov, Assanali Abu, Kok-Seng Wong, Min-Ho LeeIEEE Access 2024 · DOI

TL;DR

This paper introduces FedFSLAR++, a novel federated learning framework for few-shot action recognition. It addresses the challenges of data privacy, limited labeled data, and the need for robust spatiotemporal feature extraction in video analysis by benchmarking various 3D CNNs, Transformers, and hybrid models within a federated setting.

What Problem Does This Paper Address?

Human action recognition (HAR) is crucial for applications like surveillance and video analysis. However, training effective HAR models requires vast amounts of labeled video data, which raises privacy concerns and incurs significant storage and communication costs. Few-shot learning (FSL) aims to address data scarcity by enabling models to learn new actions from very few examples. Federated learning (FL) offers a privacy-preserving approach by training models collaboratively across decentralized devices without sharing raw data.

This paper tackles the intersection of these challenges: Federated Few-Shot Action Recognition (FedFSLAR). Existing FSL methods often rely on 2D features, neglecting crucial temporal dynamics in videos, and are typically centralized, posing privacy risks. While FL addresses privacy, applying it to FSL for complex tasks like action recognition, especially with heterogeneous data distributions across clients, remains underexplored. This work aims to establish a unified benchmark to systematically evaluate and compare different components of FedFSLAR, including feature extraction, meta-learning algorithms, and FL aggregation strategies.

Methods

The core of this work is the FedFSLAR++ framework, which integrates federated learning with few-shot learning and advanced feature extraction for video-based action recognition.

  • Federated Learning (FL): The framework utilizes FL to train a global model collaboratively across multiple clients, preserving data privacy. It benchmarks three popular FL algorithms: FedAvg, FedProx, and MOON.
  • Few-Shot Learning (FSL): The goal is to train models that can recognize new action classes with only a few labeled examples. The paper explores three meta-learning paradigms: Reptile (optimization-based), ProtoNet (metric-based), and RelationNet (metric-based).
  • Feature Extraction: A key contribution is the benchmarking of various spatiotemporal feature extractors. This includes:
  • 3D CNNs: I3D, R3D, R(2+1)D, Slow.
  • Video Transformers: VTN, MViT.
  • Hybrid Model (STCA): The authors propose STCA (Spatiotemporal Convolution with Attention Network), a novel hybrid architecture combining 3D CNNs (specifically the SlowFast pathway) with Transformer's self-attention mechanism to leverage the strengths of both. They also introduce an attention-based RelationNet tailored for compatibility with Transformer outputs.
  • Benchmarking: The paper establishes a systematic benchmark to compare these components under various FL settings (IID and non-IID data distributions) and with or without pre-training.

Datasets

The experiments were conducted on four widely-used benchmark datasets for action recognition:

  • Kinetics (K100): A subset of Kinetics 400, focusing on everyday activities.
  • Something-Something v2 (SSv2): Known for its focus on object interactions and temporal dynamics, making it challenging.
  • UCF101: A dataset with a mix of appearance and scene understanding, and higher inter-class variability.
  • HMDB51: A smaller dataset with diverse human activities, including facial and full-body actions.

The paper uses standard splits for these datasets and considers both IID (Identical and Independent Distribution) and non-IID (Non-Identical and Independent Distribution) data partitions to simulate realistic FL scenarios.

Key Insights and Ideas

  • Spatiotemporal features are crucial for FSL: Unlike many prior works relying on 2D features, this paper strongly advocates for 3D CNNs and Video Transformers to capture the temporal dynamics essential for effective action recognition, especially in few-shot scenarios.
  • Hybrid models (STCA) offer a strong balance: The proposed STCA model, combining CNNs and Transformers, consistently achieves state-of-the-art performance, outperforming pure CNNs and Transformers, particularly when pre-trained. It also offers a good balance between accuracy and computational complexity.
  • Pre-training is a game-changer for FedFSL: Leveraging pre-trained feature extractors significantly boosts FSL performance, especially for Transformer-based models. This highlights the importance of transferring knowledge from large external datasets.
  • Federated learning enhances FSL: In many scenarios, especially with motion-centric datasets like SSv2, federated learning not only preserves privacy but also leads to better or comparable accuracy compared to centralized training, demonstrating its ability to leverage diverse data.
  • Metric-based meta-learners (ProtoNet, RelationNet) are better suited for FedFSL: Compared to optimization-based methods like Reptile, metric-based approaches show superior performance in federated few-shot action recognition, with RelationNet showing particular robustness in FL settings.

Key Results

  • STCA outperforms other extractors: The proposed STCA model achieved the highest accuracy on most datasets and settings, especially when pre-trained. For example, on K100, pre-trained R(2+1)D achieved 90.74% (5-shot, non-IID), while pre-trained STCA achieved 76.7% (5-shot, non-IID) without pre-training, and significantly higher with pre-training.
  • Pre-trained MViT and STCA excel: When pre-trained, MViT and STCA consistently outperformed pre-trained CNN models by a large margin, achieving accuracies like 99.38% (5-shot, non-IID) for MViT on UCF101.
  • FL improves performance: For SSv2, FL settings consistently yielded better accuracy than CL, with pre-trained STCA achieving 77.82% (5-shot, IID) in FL versus 76.48% (5-shot, IID) in CL.
  • FedProx is a strong FL algorithm: FedProx consistently outperformed FedAvg and MOON in many scenarios, especially in non-IID settings.
  • Model complexity vs. performance: STCA offers a good trade-off, achieving high accuracy with moderate FLOPs compared to more complex models like MViT.

Limitations and Future Work

  • Computational demands of FL: The paper acknowledges that FL can impose excessive computational demands, especially with large backbone models, making it challenging for resource-constrained edge devices.
  • Future work directions:
    • Knowledge distillation: To improve efficiency for edge devices, knowledge distillation can be used to transfer knowledge from large central models to lightweight models.
  • Large-scale datasets: Evaluating the framework on very large-scale datasets like YouTube-8M to understand universality and applicability.
  • Self-supervised pre-training: Developing efficient self-supervised pre-training methods within FL to learn effective video representations from unlabeled data.
  • Multimodal fusion: Incorporating additional modalities like human skeleton data and text to further boost recognition performance.
  • Realistic datasets: Creating more complex and realistic action datasets to enhance generalizability.

Summary updated

BibTeX
@article{tu2024benchmarking,
  author = {Nguyen Anh Tu and Nartay Aikyn and Nursultan Makhanov and Assanali Abu and Kok-Seng Wong and Min-Ho Lee},
  title = {{Benchmarking Federated Few-Shot Learning for Video-Based Action Recognition}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3519254},
}
Figure