Expanding Sparse Labels for Action Recognition

42 concepts41 papers2 proposals

The Semantic Poverty of Discrete Labels

Traditional action and activity recognition models operate under a significant constraint: they map complex, high-dimensional visual or physical sequences to short, discrete categorical labels. A single word or short phrase like "drinking," "slicing," or "jumping" is expected to serve as the sole anchor for a vast array of visual configurations. This creates a severe semantic gap, often referred to as label sparsity or class-name underspecification.

When a model is trained only on isolated class names, it lacks the descriptive depth required to understand the underlying mechanics of an activity. As noted by Shreyank N Gowda et al., simple labels fail to capture the nuances of motion and interaction, making it exceptionally difficult to map visual temporal patterns to meaningful semantic descriptions. This sparse representation forces models to memorize superficial shortcuts—such as static background objects—rather than learning the actual temporal logic or physical dynamics of an action (Chinmaya Devaraj et al.; the authors).

Leveraging Dense Textual Landscapes

Why does expanding a single-word label into a detailed natural language description fundamentally change how a model learns? The answer lies in the structure of modern vision-language foundation models like CLIP.

When we use a single-word embedding as a semantic bridge, the resulting representation is often too sparse or vague to capture distinct visual characteristics, leading to severe semantic ambiguity (Yannick Le Cacheux et al.; Shizhe Chen et al.). However, foundation models are trained on millions of rich, descriptive web captions. They possess a dense, continuous semantic space where detailed sentences are already mapped to complex visual concepts.

By transforming a sparse label into an explicit textual description of the action's sub-steps, tools, and context, we "piggyback" on this pre-existing knowledge. Instead of forcing a model to learn a direct mapping from a video to an isolated, arbitrary token, we align the video with a descriptive sentence that the foundation model already understands. This shift from rigid classification to flexible natural-language description enables robust open-vocabulary and zero-shot generalization (Fakhri Karray).

LLM-Driven Label Expansion

The most prominent mechanism for overcoming label sparsity is using Large Language Models (LLMs) to automatically generate descriptive context during training or inference.

Instead of relying on manual annotations, frameworks leverage LLMs to "explain" actions before identifying them. For example, Chengyou Jia et al. use LLMs to generate multi-attribute descriptions detailing the specific tools used and body movements involved. Similarly, Massimo Bosetti et al. expand simple action labels into rich descriptions of sub-actions and environmental context, allowing frozen vision-language models to bridge the gap between static images and temporal video complexity without additional training.

This approach provides the necessary visual and behavioral context to differentiate between visually similar activities that share broad category labels but differ in fine-grained execution (Akshita Gupta et al.; Geuntaek Lim et al.). By enriching text labels with LLM-generated context, models can effectively map physical movements to conceptual categories even for entirely unseen classes (Hongsong Wang et al.; Rui Yan et al.).

Grounding Actions in Physical and Spatial Primitives

Another facet of this research direction focuses on decomposing monolithic action labels into fine-grained physical mechanics, human-object interactions, or atomic body states.

Rather than mapping pixels directly to a high-level label, some frameworks adopt a neuro-symbolic approach. Yong-Lu Li et al. introduce a method that first identifies atomic body part states ("primitives") and then uses logic rules to compose them into complex activities, overcoming the low semantic coverage typical of standard action datasets. For human-object interactions, Yichao Cao et al. utilize spatial locations as prompts alongside LLM-generated descriptive text to capture the underlying physical mechanics of an interaction, while Weibo Jiang models the complex dependencies and correlations between different interactions within a scene to handle visual diversity.

This granular decomposition is especially critical in egocentric (first-person) video understanding and complex event analysis, where high-level labels completely omit the fine-grained details of hand movements, object states, and temporal event flows necessary for accurate interpretation or high-fidelity synthesis (Bolin Lai et al.; Jiayi Zou et al.).

Beyond Video: Motion Sequences and Wearable Sensors

The problem of semantic label sparsity extends beyond standard video datasets into 3D human motion generation and sensor-based Human Activity Recognition (HAR).

In 3D motion analysis, datasets frequently suffer from simplistic, single-word labels or raw spatial trajectories that lack domain-specific semantic granularity (Ming Yan et al.; Xinghan Wang et al.). To enable models to interpret complex or creative instructions, Guy Tevet et al. align a 3D motion auto-encoder's latent space directly with CLIP, mapping abstract textual concepts to precise physical dynamics.

In the domain of wearable sensors, the "textual paucity" of brief categorical labels prevents models from understanding the underlying anatomical or biomechanical mechanisms of an activity through natural language supervision (Harish Haresamudram et al.). To resolve this, Diyan Teng demonstrates that augmenting raw sensor data with explicit "sensor context"—textual descriptions of the sensor's type, axis, and body placement—provides the necessary descriptive depth to allow effective zero-shot transfer to entirely unseen activities.

Key Foundational Work

The KIT Motion-Language Dataset (2016) — contributed 1 concept elements to this direction.
Weakly-Supervised Alignment of Video with Text (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_label_sparsity_21b1ab18,
  title = {semantic label sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-label-sparsity/rdr_21b1ab18c4784dbfa7154ac4c8813edb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_label_sparsity_21b1ab18_v1,
  title = {semantic label sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-label-sparsity/rdr_21b1ab18c4784dbfa7154ac4c8813edb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake