Introduction
Human Activity Recognition (HAR) is a foundational technology for service robotics, particularly in elderly care and smart home automation. However, domestic environments are inherently "noisy," characterized by clutter, occlusions, and unpredictable lighting. A significant challenge in this domain is fine-grained recognition—distinguishing between visually similar actions such as "brushing teeth" and "applying lipstick."
Traditional approaches often rely on specialized hardware like depth sensors or skeleton trackers, which may not generalize well to real-world home settings. KRAST (Knowledge-Augmented Robotic Action Recognition with Structured Text) addresses this by leveraging the "common sense" embedded in large Vision-Language Models (VLMs) like CLIP, enriched with structured, domain-specific textual context to improve discriminative power.
Methodology
KRAST is built upon a frozen, pre-trained VLM backbone. To maintain efficiency and preserve general knowledge, the researchers utilized Prompt Learning rather than full model fine-tuning. The architecture consists of two primary components: Segmented Knowledge Prompt Tuning (SegKPT) and the Video Prompt Learner (VitaVPL).
Segmented Knowledge Prompt Tuning (SegKPT)
The core innovation of KRAST is the use of structured text prompts. Instead of simple labels, the model is provided with three layers of knowledge:
- Hierarchical (H): Places the action in a broad category (e.g., "Personal Care").
- Semantic (S): Provides a concise description of the motion (e.g., "rubbing palms together with soap").
- Discriminative (D): Explicitly highlights unique features that distinguish the action from visually similar ones.
Video Prompt Learner (VitaVPL)
To capture the temporal evolution of actions, the Video Prompt Learner (VitaVPL) inserts learnable "summary," "global," and "local" tokens into the Vision Transformer (ViT) layers. This allows the model to better understand how an action unfolds across multiple frames over time.
Experimental Setup and Results
The researchers evaluated KRAST on the ETRI-Activity3D dataset, a large-scale robotics benchmark featuring 55 daily activity classes performed by both elderly and young subjects. Generalization was also tested on the NTU RGB+D 60 dataset.
KRAST achieved a state-of-the-art 95.22% Top-1 accuracy on the ETRI-Activity3D dataset. Notably, the model outperformed existing methods that utilize multiple data modalities (such as depth or skeleton data) while using only standard RGB video at test time. Ablation studies confirmed that the structured prompting strategy alone accounted for a nearly 20% improvement in accuracy over the baseline.
Key Insights
The "Temporal Sweet Spot"
An important finding in the study is that more data does not always lead to better performance. The model's accuracy peaked at 32 frames per video clip. Increasing the frame count further (up to 86) caused performance to plateau or dip, likely because redundant frames introduced noise that distracted the model's attention mechanism.
Latent Space Separation
Visualizing text embeddings showed that before fine-tuning, many semantically distinct actions were clumped together in the model's latent space. KRAST's prompt tuning successfully moved these actions further apart, creating a clearer "taxonomy of motion" that facilitates more accurate classification.
Conclusion and Future Work
KRAST demonstrates that enriching Vision-Language Models with structured textual knowledge significantly enhances robotic action recognition in complex environments. While highly effective, future work will focus on optimizing the pipeline for real-time deployment on mobile robots and exploring continual learning techniques that allow robots to learn new actions through natural language interaction on the fly.
Summary updated