Progressive Learning of Instance-Level Proxy Semantics for Few-Shot Action Recognition

Fang Peng, Xiaoshan Yang, Yaowei Wang, Changsheng Xu

Abstract

Few-shot action recognition is a crucial task for mitigating the challenges of data scarcity in video understanding. Recent advancements in large-scale pre-trained models have introduced the potential of incorporating semantic knowledge from multi-modal pre-trained models, such as CLIP, to alleviate these challenges. Although some progress have been made, existing methods still rely on class-level text embeddings that are inherently low in diversity, limiting their ability to generalize to unseen actions. To overcome this limitation, we propose a novel framework called Progressive Learning of Instance-Level Proxy Semantics (ProLIPS). ProLIPS integrates Proxy Semantic Diffusion (PSD) to generate rich, instance-level proxy semantic features with diverse semantic contents and temporal dynamics, utilizing a multi-step CLIP-guidance mechanism and a time-conditioned reverse diffusion process. Our approach preserves the diversity of semantic-aligned visual features, significantly improving the generalization and robustness of few-shot action recognition. Extensive experiments on five challenging benchmarks demonstrate the effectiveness of ProLIPS.
BibTeX
@article{peng2026progressive,
  author = {Fang Peng and Xiaoshan Yang and Yaowei Wang and Changsheng Xu},
  title = {{Progressive Learning of Instance-Level Proxy Semantics for Few-Shot Action Recognition}},
  journal = {IEEE Transactions on Multimedia},
  year = {2026},
  url = {https://openreview.net/forum?id=PX1I3mGvGO},
}

Loading related content...

Figure