SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition

Hongda Liu, Yunfan Liu, Changlu Wang, Yunlong Wang, Zhenan SunarXiv 2025 · DOI · arXiv

Introduction

Skeleton-based action recognition is a critical task in computer vision, valued for its computational efficiency and robustness to environmental factors like lighting. Recent advancements have leveraged Large Language Models (LLMs) to provide semantic priors, enriching skeletal data with textual descriptions. However, current methods often suffer from two primary limitations: homogeneous semantics, where similar actions receive nearly identical descriptions, and coarse alignment, where global features are matched without fine-grained joint-level constraints.

Comparison between (a) traditional one-turn prompting and (b) the proposed agentic interaction framework.

To address these challenges, we introduce SkeletonAgent, a novel framework that shifts from static, one-way prompting to a dynamic, agentic interaction. By incorporating a feedback loop, SkeletonAgent iteratively refines action descriptions and skeletal focus, achieving state-of-the-art performance across multiple benchmarks.

Methodology: The Agentic Interaction Loop

The core of SkeletonAgent lies in its two cooperative agents: the Questioner Agent and the Selector Agent. This duo transforms the recognition process into a continuous dialogue between the model's performance and the LLM's semantic knowledge.

Detailed architecture of the SkeletonAgent framework, illustrating the iterative feedback loop between the Questioner and Selector agents.

The Questioner Agent (Context-aware Prompter)

The Questioner Agent monitors the model's training progress via an online confusion matrix. When the model consistently misclassifies certain actions (e.g., confusing "drinking" with "brushing teeth"), the Questioner generates targeted prompts for the LLM. Instead of generic descriptions, it asks for specific discriminative features that distinguish the confused classes. This "Targeted Distinction" approach ensures the semantic priors are contextually relevant and highly discriminative.

The Selector Agent (Adaptive Knowledge Highlighter)

The Selector Agent processes the LLM's responses to extract actionable constraints for the skeleton encoder:

  1. Local Joint Constraints: It identifies critical joints mentioned in the text (e.g., the wrist and elbow for "hand waving") and constructs a topology matrix ($K$). This matrix guides the model to prioritize these joints during feature extraction.
  2. Global Semantic Alignment: It aligns the extracted skeletal features with the specific discriminative cues provided by the LLM, ensuring that the model learns the subtle nuances that separate similar actions.

The framework is optimized using a multi-task loss function that combines standard classification with a constraint loss ($L_{con}$) to enforce joint-level focus and an alignment loss ($L_{align}$) to synchronize text and motion features.

Experimental Evaluation

SkeletonAgent was evaluated on five major benchmarks: NTU RGB+D (60 & 120), Kinetics-Skeleton, FineGYM, and UAV-Human.

Performance Highlights

  • State-of-the-Art Results: The framework achieved 94.5% accuracy on the NTU-60 Cross-Subject split and a remarkable 96.5% on FineGYM, which focuses on fine-grained gymnastic movements.
  • Hard Class Improvement: On challenging actions with low baseline accuracy (<70%), SkeletonAgent provided a 4.3% performance boost over standard Graph Convolutional Networks (GCNs).
  • LLM Agnostic: While GPT-4o yielded the best results, the framework showed consistent gains across various LLMs (Gemini, DeepSeek), indicating that the agentic architecture itself is the primary driver of improvement.

Discussion and Conclusion

SkeletonAgent demonstrates that moving from static semantic priors to an agentic feedback loop significantly enhances the discriminative power of skeleton-based action recognition models. By identifying "why" a model fails and providing targeted semantic guidance, the system effectively bridges the gap between high-level text and low-level skeletal motion.

While the framework introduces some complexity during training, the benefits in accuracy and robustness are substantial. Future work will explore reducing inference latency and extending this agentic paradigm to other multimodal tasks such as video-to-text generation and robotic control.

Summary updated

Abstract

Recent advances in skeleton-based action recognition increasingly leverage semantic priors from Large Language Models (LLMs) to enrich skeletal representations. However, the LLM is typically queried in isolation from the recognition model and receives no performance feedback. As a result, it often fails to deliver the targeted discriminative cues critical to distinguish similar actions. To overcome these limitations, we propose SkeletonAgent, a novel framework that bridges the recognition model and the LLM through two cooperative agents, i.e., Questioner and Selector. Specifically, the Questioner identifies the most frequently confused classes and supplies them to the LLM as context for more targeted guidance. Conversely, the Selector parses the LLM's response to extract precise joint-level constraints and feeds them back to the recognizer, enabling finer-grained cross-modal alignment. Comprehensive evaluations on five benchmarks, including NTU RGB+D, NTU RGB+D 120, Kinetics-Skeleton, FineGYM, and UAV-Human, demonstrate that SkeletonAgent consistently outperforms state-of-the-art benchmark methods. The code is available at https://github.com/firework8/SkeletonAgent.
BibTeX
@misc{liu2025skeletonagent,
  author = {Hongda Liu and Yunfan Liu and Changlu Wang and Yunlong Wang and Zhenan Sun},
  title = {{SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition}},
  year = {2025},
  eprint = {2511.22433},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.22433},
}
Figure