Unlocking Multi-Site Clinical Data: A Federated Approach to Privacy-First Child Autism Behavior Analysis

Guangyu Sun, Wenhan Wu, Zhishuai Guo, Ziteng Wang, Pegah Khosravi, Chen ChenarXiv 2026 · arXiv

TL;DR

This paper presents the first federated learning framework for pose-based child autism behavior recognition, addressing the critical conflict between data privacy and collaborative clinical research. By combining 3D skeletal abstraction with Adaptive Personalized Federated Learning (APFL), the framework enables multiple clinical sites to collaboratively train robust models without sharing sensitive raw video data, successfully overcoming severe cross-site data heterogeneity.

What Problem Does This Paper Address?

Automated recognition of autism-related behaviors in children offers a powerful, objective tool to complement traditional clinical assessments, which are often prone to human observer bias and high resource demands. However, deploying deep learning at scale in this domain faces a massive roadblock: privacy regulations.

Under strict frameworks like HIPAA and GDPR, raw RGB video recordings of children are considered primary biometric identifiers containing Protected Health Information (PHI). Consequently, medical institutions cannot easily centralize or share these datasets, leaving valuable clinical data trapped in isolated "data silos." Furthermore, individual clinics often suffer from data scarcity and severe data heterogeneity (Non-IID data) caused by variations in therapeutic protocols, room layouts, camera angles, and patient demographics. Standard machine learning models fail to generalize across these diverse environments without a privacy-preserving way to collaborate.

Methods

The authors propose a robust, privacy-first framework built on a two-layer privacy protection strategy:

  1. Layer 1: Skeletal Abstraction: Instead of raw RGB video, the framework extracts 3D skeletal representations of the human body. This filters out identifiable facial features, clothing, and environmental contexts, decoupling critical motion patterns from visual identity.
  2. Layer 2: Decentralized Optimization (Federated Learning): The abstracted pose sequences remain localized within each clinic's secure infrastructure. Only anonymized model parameters are shared with a central aggregator.
Figure 1. Overview of the proposed Two-Layer Privacy framework. The first layer achieves privacy via skeletal abstraction, filtering out raw biometric identifiers from videos. This is done at each clinical site for their local video data (a demonstration is shown for ‘Clinical Site A’ in this figure). The second layer employs decentralized optimization (FL) using the efficient FreqMixFormer backbone to maintain data residency. Adaptive personalization layers (FedBN, FedPer, or APFL) are evaluated to handle clinical heterogeneity by adaptively mixing global knowledge with local specialization.

Model Architecture & Personalization

  • Backbone: The framework utilizes FreqMixFormer, a state-of-the-art skeleton-based action recognition model. It leverages a frequency-aware attention module via the Discrete Cosine Transform (DCT) to capture discriminative temporal motion patterns efficiently, making it ideal for resource-constrained clinical edge nodes.
  • Personalized Federated Learning (PFL): To handle cross-site domain shifts, the authors systematically evaluate several PFL strategies (FedBN, FedPer, and APFL). Adaptive Personalized Federated Learning (APFL) emerged as the core strategy, which dynamically learns an optimal mixing ratio ($\alpha$) between the global collaborative model and a localized model for each site.

Datasets

The framework was evaluated using a simulated multi-site benchmark derived from the MultiModal ASD (MMASD) benchmark.

  • Dataset Details: MMASD contains 1,315 segmented samples from over 100 hours of intervention recordings involving 32 children diagnosed with Autism Spectrum Disorder (ASD). It features 71 body keypoints extracted via the ROMP algorithm.
  • Multi-site Partitioning: To simulate a realistic cross-silo federated environment, the dataset was partitioned into three distinct clients based on clinical therapeutic themes:
  • Theme 1: Robotic-assisted therapy (e.g., Arm Swing, Squat)
  • Theme 2: Rhythm-based activities (e.g., Drumming, Sing and Clap)
  • Theme 3: Yoga-based poses (e.g., Frog Pose, Tree Pose)

Key Insights and Ideas

  • The Failure of Monolithic Global Models: Standard federated learning (FedAvg) performed significantly worse than purely independent local training. This reveals that simple model aggregation is counterproductive in autism behavior recognition because severe cross-site domain shifts mask universal behavioral markers.
  • Holistic vs. Layer-wise Personalization: Restricting personalization to specific layers (like local batch normalization in FedBN or local classifier heads in FedPer) yielded unstable training dynamics. For high-dimensional spatiotemporal data like 3D skeletons, adaptation must occur holistically across the entire model representation.
  • The "Knowledge Anchor" Effect: Analysis of APFL's adaptive mixing parameter ($\alpha$) showed that the model initially prioritizes global knowledge synthesized from the collaborative network before gradually shifting focus toward site-specific nuances. The global model acts as a critical anchor that stabilizes local learning, especially for sites with limited data.

Key Results

The quantitative evaluation highlights the clear advantage of adaptive personalization:

  • APFL achieved the highest average recognition accuracy of 87.80%, outperforming the independent local training baseline (82.61%) and drastically outperforming standard FedAvg (70.30%).
  • In Theme 1 (Robotic-assisted therapy), APFL achieved a peak accuracy of 92.74%, recovering a massive 17% performance drop suffered by standard FedAvg due to clinical heterogeneity.
Figure 3. Performance evolution across 30 communication rounds for different clinical themes. The curves demonstrate the conver- gence characteristics of standard FL, parameter-wise PFL, and our adaptive personalization approach. APFL (red) consistently achieves superior stability and final recognition accuracy across all therapeutic domains.

Limitations and Future Work

While the framework successfully secures visual and biometric privacy, it is currently limited to motion-only analysis via 3D skeletons. The authors note that autism spectrum disorder manifests across multiple modalities that this model cannot yet capture.

Future Directions: 1. Multimodal Integration: Incorporating audio streams to analyze speech patterns, vocal prosody, and conversational dynamics between patients and clinicians. 2. Federated LLMs: Utilizing privacy-preserving Large Language Models (LLMs) to analyze clinical notes and transcripts alongside behavioral data in a secure, decentralized manner.

Summary updated

Abstract

Automated recognition of autistic behaviors in children is essential for early intervention and objective clinical assessment. However, the development of robust models is severely hindered by strict privacy regulations (e.g., HIPAA) and the sensitive nature of pediatric data, which prevents the centralized aggregation of clinical datasets. Furthermore, individual clinical sites often suffer from data scarcity, making it difficult to learn generalized behavior patterns or tailor models to site-specific patient distributions. To address these challenges, we observe that Federated Learning (FL) can decouple model training from raw data access, enabling multi-site collaboration while maintaining strict data residency. In this paper, we present the first study exploring Federated Learning for pose-based child autism behavior recognition. Our framework employs a two-layer privacy protection mechanism: utilizing human skeletal abstraction to remove identifiable visual information from the raw RGB videos and FL to ensure sensitive pose data remains within the clinic. This approach leverages distributed clinical data to learn generalized representations while providing the flexibility for site-specific personalization. Experimental results on the MMASD benchmark demonstrate that our framework achieves high recognition accuracy, outperforming traditional federated baselines and providing a robust, privacy-first solution for multi-site clinical analysis.
BibTeX
@misc{sun2026unlocking,
  author = {Guangyu Sun and Wenhan Wu and Zhishuai Guo and Ziteng Wang and Pegah Khosravi and Chen Chen},
  title = {{Unlocking Multi-Site Clinical Data: A Federated Approach to Privacy-First Child Autism Behavior Analysis}},
  year = {2026},
  eprint = {2604.02616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.02616},
}
Figure