PromptGAR: Flexible Promptive Group Activity Recognition

Zhangyu Jin, Andrew Feng, Ankur Chemburkar, Celso M. de Melo

Abstract

We present PromptGAR, a novel framework that addresses the limitations of current Group Activity Recognition (GAR) approaches by leveraging multi-modal prompts to achieve both input flexibility and high recognition accuracy. The existing approaches suffer from limited real-world applicability due to their reliance on full prompt annotations, the lack of long-term actor consistency, and under-exploration of multi-group scenarios. To bridge the gap, we proposed PromptGAR, which is the first GAR model to provide input flexibility across prompts, frames, and instances without the need for retraining. Specifically, we unify bounding boxes, skeletal keypoints, and areas as point prompts and employ a recognition decoder for cross-updating class and prompt tokens. To ensure long-term consistency for extended activity durations, we also introduce a relative instance attention mechanism that directly encodes instance IDs. Finally, PromptGAR explores the use of area prompts to enable the selective recognition of the particular group activity within videos that contain multiple concurrent groups. Comprehensive evaluations demonstrate that PromptGAR achieves competitive performances both on full prompts and diverse prompt inputs, establishing its effectiveness on input flexibility and generalization ability for real-world applications.

Introduction

Group Activity Recognition (GAR) involves identifying collective actions in video, such as a "spike" in volleyball or a "3-point shot" in basketball. While deep learning has advanced this field, existing models suffer from three major "rigidity" problems:

  1. Annotation Dependency: Most models require "full prompts" (perfect bounding boxes and skeletal data) at test time. In the real world, detectors fail, IDs switch, and annotations are often incomplete.
  2. Fixed Input Shapes: Many architectures are hard-coded to process a specific number of frames or a fixed number of actors. If a spectator is accidentally detected as a player, or a player leaves the frame, these models often break or lose significant accuracy.
  3. Lack of Actor Consistency: Many models rely on a fixed "player order" in their data tensors. If you shuffle the list of players, the model’s prediction might change, which is a fundamental flaw for real-world deployment where player ordering is arbitrary.

Methodology

The authors introduce PromptGAR, which treats GAR as a "promptable" task, similar in spirit to the Segment Anything Model (SAM). Unlike traditional methods, PromptGAR provides input flexibility across prompts, frames, and instances without the need for retraining.

Point Prompt Unification

Instead of treating bounding boxes and skeletons as different data structures, PromptGAR unifies them as Point Prompts. A bounding box is represented by three points (top-left, center, bottom-right), and a skeleton by 17 keypoints. Each point carries attributes: spatial coordinates $(x, y)$, temporal location $(t)$, point type $(p)$, and instance ID.

PromptGAR unifies bounding boxes and skeletons as point prompts with instance IDs to handle diverse input types.

Relative Instance Attention

To solve the "player order" problem, the authors replaced absolute identity encoding with a relative mechanism. The attention block specifically encodes whether two tokens belong to the same individual or different individuals. This ensures the model is permutation-invariant—it doesn't matter what order the players are listed in.

Experimental Evaluation

The model was evaluated on two standard GAR benchmarks: the Volleyball Dataset (4,830 clips) and the NBA Dataset (9,172 clips). The NBA dataset is particularly challenging due to longer sequences (72 frames) and complex activities (e.g., "2p-layup-fail").

Key Results

PromptGAR achieved 96.0% Top-1 accuracy on the Volleyball dataset and 80.6% on the NBA dataset, surpassing previous leaders like GroupFormer and Flaming-Net.

PromptGAR achieves the best balance between input flexibility and accuracy compared to existing methods on the Volleyball dataset.

One of the most impressive findings is that PromptGAR can be trained on "full" annotations but performs remarkably well when prompts are partially or entirely removed at test time. It degrades gracefully rather than failing. On the Volleyball dataset, reducing the number of tracked instances from 12 down to just 3 resulted in only a 1.7% drop in accuracy. Even with zero prompts (just the raw video), PromptGAR maintained 93.9% accuracy, outperforming several older models that had access to full bounding boxes.

Limitations and Future Work

Due to the memory intensity of processing long video sequences with transformers, the model is currently limited to a 224 × 224 input resolution. Future work could involve integrating more diverse prompts, such as natural language descriptions or "ball trajectories," to further enhance the model's reasoning capabilities in complex sports scenarios.

Summary updated

Abstract

We present PromptGAR, a novel framework that addresses the limitations of current Group Activity Recognition (GAR) approaches by leveraging multi-modal prompts to achieve both input flexibility and high recognition accuracy. The existing approaches suffer from limited real-world applicability due to their reliance on full prompt annotations, the lack of long-term actor consistency, and under-exploration of multi-group scenarios. To bridge the gap, we proposed PromptGAR, which is the first GAR model to provide input flexibility across prompts, frames, and instances without the need for retraining. Specifically, we unify bounding boxes, skeletal keypoints, and areas as point prompts and employ a recognition decoder for cross-updating class and prompt tokens. To ensure long-term consistency for extended activity durations, we also introduce a relative instance attention mechanism that directly encodes instance IDs. Finally, PromptGAR explores the use of area prompts to enable the selective recognition of the particular group activity within videos that contain multiple concurrent groups. Comprehensive evaluations demonstrate that PromptGAR achieves competitive performances both on full prompts and diverse prompt inputs, establishing its effectiveness on input flexibility and generalization ability for real-world applications.
BibTeX
@inproceedings{jin2026promptgar,
  author = {Zhangyu Jin and Andrew Feng and Ankur Chemburkar and Celso M. de Melo},
  title = {{PromptGAR: Flexible Promptive Group Activity Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
}
Figure