Introduction
Skeleton-based action recognition is a critical task in computer vision, valued for its computational efficiency and robustness to environmental factors like lighting. Recent advancements have leveraged Large Language Models (LLMs) to provide semantic priors, enriching skeletal data with textual descriptions. However, current methods often suffer from two primary limitations: homogeneous semantics, where similar actions receive nearly identical descriptions, and coarse alignment, where global features are matched without fine-grained joint-level constraints.
To address these challenges, we introduce SkeletonAgent, a novel framework that shifts from static, one-way prompting to a dynamic, agentic interaction. By incorporating a feedback loop, SkeletonAgent iteratively refines action descriptions and skeletal focus, achieving state-of-the-art performance across multiple benchmarks.
Methodology: The Agentic Interaction Loop
The core of SkeletonAgent lies in its two cooperative agents: the Questioner Agent and the Selector Agent. This duo transforms the recognition process into a continuous dialogue between the model's performance and the LLM's semantic knowledge.
The Questioner Agent (Context-aware Prompter)
The Questioner Agent monitors the model's training progress via an online confusion matrix. When the model consistently misclassifies certain actions (e.g., confusing "drinking" with "brushing teeth"), the Questioner generates targeted prompts for the LLM. Instead of generic descriptions, it asks for specific discriminative features that distinguish the confused classes. This "Targeted Distinction" approach ensures the semantic priors are contextually relevant and highly discriminative.
The Selector Agent (Adaptive Knowledge Highlighter)
The Selector Agent processes the LLM's responses to extract actionable constraints for the skeleton encoder:
- Local Joint Constraints: It identifies critical joints mentioned in the text (e.g., the wrist and elbow for "hand waving") and constructs a topology matrix ($K$). This matrix guides the model to prioritize these joints during feature extraction.
- Global Semantic Alignment: It aligns the extracted skeletal features with the specific discriminative cues provided by the LLM, ensuring that the model learns the subtle nuances that separate similar actions.
The framework is optimized using a multi-task loss function that combines standard classification with a constraint loss ($L_{con}$) to enforce joint-level focus and an alignment loss ($L_{align}$) to synchronize text and motion features.
Experimental Evaluation
SkeletonAgent was evaluated on five major benchmarks: NTU RGB+D (60 & 120), Kinetics-Skeleton, FineGYM, and UAV-Human.
Performance Highlights
- State-of-the-Art Results: The framework achieved 94.5% accuracy on the NTU-60 Cross-Subject split and a remarkable 96.5% on FineGYM, which focuses on fine-grained gymnastic movements.
- Hard Class Improvement: On challenging actions with low baseline accuracy (<70%), SkeletonAgent provided a 4.3% performance boost over standard Graph Convolutional Networks (GCNs).
- LLM Agnostic: While GPT-4o yielded the best results, the framework showed consistent gains across various LLMs (Gemini, DeepSeek), indicating that the agentic architecture itself is the primary driver of improvement.
Discussion and Conclusion
SkeletonAgent demonstrates that moving from static semantic priors to an agentic feedback loop significantly enhances the discriminative power of skeleton-based action recognition models. By identifying "why" a model fails and providing targeted semantic guidance, the system effectively bridges the gap between high-level text and low-level skeletal motion.
While the framework introduces some complexity during training, the benefits in accuracy and robustness are substantial. Future work will explore reducing inference latency and extending this agentic paradigm to other multimodal tasks such as video-to-text generation and robotic control.
Summary updated