SelaFD:Seamless Adaptation of Vision Transformer Fine-tuning for Radar-based Human Activity

Yijun Wang, Yong Wang, Changsheng Xu, Yao Shuai, Qisong WuarXiv 2025 · DOI · arXiv

Abstract

Human Activity Recognition (HAR) such as fall detection has become increasingly critical due to the aging population, necessitating effective monitoring systems to prevent serious injuries and fatalities associated with falls. This study focuses on fine-tuning the Vision Transformer (ViT) model specifically for HAR using radar-based Time-Doppler signatures. Unlike traditional image datasets, these signals present unique challenges due to their non-visual nature and the high degree of similarity among various activities. Directly fine-tuning the ViT with all parameters proves suboptimal for this application. To address this challenge, we propose a novel approach that employs Low-Rank Adaptation (LoRA) fine-tuning in the weight space to facilitate knowledge transfer from pre-trained ViT models. Additionally, to extract fine-grained features, we enhance feature representation through the integration of a serial-parallel adapter in the feature space. Our innovative joint fine-tuning method, tailored for radar-based Time-Doppler signatures, significantly improves HAR accuracy, surpassing existing state-of-the-art methodologies in this domain.

Introduction

Human Activity Recognition (HAR) is a cornerstone of modern healthcare monitoring, particularly for fall detection in elderly populations. While wearables and cameras are prevalent, they suffer from compliance issues and privacy concerns. Millimeter-wave radar offers a non-intrusive, privacy-preserving alternative by generating Time-Doppler (TD) signatures. However, adapting powerful Vision Transformers (ViTs) to these non-visual spectrograms is challenging due to the domain gap and high similarity between certain activities, such as the subtle difference between "drinking" and "picking up an object."

Methodology

System Architecture

The proposed SelaFD framework adapts pre-trained ViTs using a dual-strategy approach. As shown in the overall pipeline, radar TD signatures are partitioned into patches, combined with position embeddings, and processed through a Transformer Encoder.

Overall pipeline for radar-based human activity recognition using Vision Transformers.

SelaFD Fine-tuning

To bridge the gap between natural image pre-training and radar data, SelaFD employs two primary mechanisms within the transformer blocks to achieve seamless adaptation:

  1. Low-Rank Adaptation (LoRA): This is applied to the Query ($Q$) and Value ($V$) projection matrices in the Multi-Head Attention (MHA) layers. By updating only these low-rank matrices while keeping the rest of the pre-trained weights frozen, the model retains "coarse-grained" knowledge from ImageNet pre-training.
  2. Serial-Parallel Adapters: To capture "fine-grained" details unique to radar, two bottleneck adapters are integrated into each block. A Serial Adapter is placed after the Attention layer to refine features sequentially, while a Parallel Adapter is placed alongside the MLP layer to enhance feature representation without disrupting the main flow.
Proposed SelaFD architecture components including LoRA and Serial-Parallel Adapters.

Key Results

The model was evaluated on the University of Glasgow (UOG) dataset, containing 1,753 TD signatures across six activities: walking, sitting, standing, drinking, picking up an object, and falling.

  • Overall Accuracy: SelaFD achieved 96.61%, significantly outperforming standard full-parameter fine-tuning (84.18%) and previous state-of-the-art methods like LH-ViT (92.10%).
  • Fall Detection: The system achieved a 100% success rate for falls, which is the most critical metric for safety-critical healthcare applications.
  • Ablation Study: Removing the adapters reduced accuracy to 95.34%, while removing LoRA dropped it to 94.07%. This confirms that the combination of weight-space and feature-space adaptation is essential for distinguishing highly similar radar signatures.

Conclusion

SelaFD demonstrates that parameter-efficient fine-tuning (PEFT) can successfully adapt large-scale vision models to specialized radar sensing tasks. By combining LoRA and custom adapters, the framework achieves superior accuracy and robustness in fall detection while maintaining computational efficiency. Future work will explore scaling this framework to larger multi-modal models and optimizing inference for real-time deployment on edge devices.

Summary updated

Abstract

Human Activity Recognition (HAR) such as fall detection has become increasingly critical due to the aging population, necessitating effective monitoring systems to prevent serious injuries and fatalities associated with falls. This study focuses on fine-tuning the Vision Transformer (ViT) model specifically for HAR using radar-based Time-Doppler signatures. Unlike traditional image datasets, these signals present unique challenges due to their non-visual nature and the high degree of similarity among various activities. Directly fine-tuning the ViT with all parameters proves suboptimal for this application. To address this challenge, we propose a novel approach that employs Low-Rank Adaptation (LoRA) fine-tuning in the weight space to facilitate knowledge transfer from pre-trained ViT models. Additionally, to extract fine-grained features, we enhance feature representation through the integration of a serial-parallel adapter in the feature space. Our innovative joint fine-tuning method, tailored for radar-based Time-Doppler signatures, significantly improves HAR accuracy, surpassing existing state-of-the-art methodologies in this domain. Our code is released at https://github.com/wangyijunlyy/SelaFD.
BibTeX
@misc{wang2025selafd,
  author = {Yijun Wang and Yong Wang and Changsheng Xu and Yao Shuai and Qisong Wu},
  title = {{SelaFD:Seamless Adaptation of Vision Transformer Fine-tuning for Radar-based Human Activity}},
  year = {2025},
  eprint = {2502.04740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.04740},
  howpublished = {arXiv},
}
Figure