Inferring social roles in long timespan video sequence

Jiangen Zhang, Wenze Hu, Benjamin Z. Yao, Yongtian Wang, Song-Chun ZhuWorkshops @ ICCV 2011 · OpenReview

Abstract

In this paper, we present a method for inferring social roles of agents (persons) from their daily activities in long surveillance video sequences. We define activities as interactions between an agent's position and semantic hotspots within the scene. Given a surveillance video, our method first tracks the locations of agents then automatically discovers semantic hotspots in the scene. By enumerating spatial/temporal locations between an agent's feet and hotspots in a scene, we define a set of atomic actions, which in turn compose sub-events and events. The numbers and types of events performed by an agent are assumed to be driven by his/her social role. With the grammar model induced by composition rules, an adapted Earley parser algorithm is used to parse the trajectories into events, sub-events and atomic actions. With probabilistic output of events, the roles of agents can be predicted under the Bayesian inference framework. Experiments are carried out on a challenging 8.5 hours video from a surveillance camera in the lobby of a research lab. Results show that our proposed method can predict the role of each agent with high precision.

Introduction

In computer vision, identifying specific actions (walking, sitting, running) is a well-established task. However, understanding the social context—the "who" behind the action—remains a significant challenge. Traditional methods for role inference often rely on visual identifiers like uniforms or badges, which are frequently absent in casual professional environments. Other approaches attempt to model person-to-person interactions, but these are often thwarted by tracking failures in crowded scenes.

This research addresses these limitations by inferring social roles purely from long-term behavior patterns and interactions with the physical environment. By focusing on how individuals engage with static "semantic hotspots" (like desks or water dispensers), the system creates a robust framework for role classification that is less sensitive to the noise of interpersonal tracking.

Methodology: Hierarchical Grammar Modeling

The core of the proposed method is a multi-layered generative model that treats human behavior as a structured language. This is achieved through a hierarchical And-Or structure:

  1. Semantic Hotspots: The system automatically identifies functional areas in a scene (entrances, exits, furniture) by clustering points where agents frequently start, stop, or linger.
  2. Atomic Actions: These are the fundamental building blocks, defined as spatial-temporal interactions with hotspots (e.g., "approaching a desk" or "leaving a sofa").
  3. Events and Sub-events: Atomic actions are composed into sub-events (Enter, Exit, Activity) which then form complete events, such as "delivering mail."
  4. Grammar Parsing: An adapted Earley parser is utilized to decompose raw movement trajectories into this hierarchical structure.
  5. Bayesian Inference: The model employs a probabilistic framework to predict roles. It uses a Poisson distribution to model the frequency of a role's appearance and a Multinomial distribution to model the proportion of different event types associated with that role.
The hierarchical And-Or tree structure used to parse raw trajectories into atomic actions, sub-events, and events.

Experimental Setup and Dataset

To evaluate the framework, the researchers collected a new benchmark: The Lobby Dataset. This dataset consists of 8.5 hours of wide-angle surveillance footage from a research lab lobby. It features 78 unique agents and 604 individual trajectories. The data is annotated with seven distinct social roles: manager, researcher, developer, engineer, staff, visitor, and mailman.

Key Insights

  • Environmental Anchoring: By prioritizing interactions with static objects over agent-to-agent interactions, the model gains significant robustness. The physical layout of the scene serves as a reliable reference point that does not change.
  • Temporal Accumulation: Role ambiguity is highest when an agent first enters the scene, as many roles share similar entry behaviors. However, the researchers observed that after approximately five events, the uncertainty (entropy) of the role prediction drops dramatically.
  • Frequency as a Discriminator: Some roles appear identical in isolated snapshots. For instance, both a "manager" and a "visitor" may approach a reception desk. The model distinguishes them by analyzing the frequency of their appearances over the 8.5-hour duration; managers appear consistently, whereas visitors are rare.

Results

The system achieved an overall role recognition accuracy of 87.2%. It was particularly effective at identifying sub-events like "Enter" (98.6%) and "Exit" (97.2%).

Confusion matrix showing the classification accuracy across the seven social roles.

As shown in the confusion matrix, roles such as "Mailman" and "Engineer" were identified with 100% accuracy. The primary areas of confusion occurred between "Managers" (73%) and "Visitors" (75%). This is attributed to the fact that visitors often follow employees, and managers frequently coordinate across multiple groups, leading to overlapping behavioral signatures.

Conclusion and Future Work

While the system demonstrates high precision, its primary limitation is generalizability. The model currently requires re-training for each new scene to map specific hotspots and role-event distributions. Future research will explore transfer learning to allow the system to apply knowledge from one functional layout (e.g., a bank lobby) to another similar environment. Additionally, integrating automated face recognition could further refine the system by removing the need for manual agent identification across disparate video segments.

Summary updated

Abstract

In this paper, we present a method for inferring social roles of agents (persons) from their daily activities in long surveillance video sequences. We define activities as interactions between an agent's position and semantic hotspots within the scene. Given a surveillance video, our method first tracks the locations of agents then automatically discovers semantic hotspots in the scene. By enumerating spatial/temporal locations between an agent's feet and hotspots in a scene, we define a set of atomic actions, which in turn compose sub-events and events. The numbers and types of events performed by an agent are assumed to be driven by his/her social role. With the grammar model induced by composition rules, an adapted Earley parser algorithm is used to parse the trajectories into events, sub-events and atomic actions. With probabilistic output of events, the roles of agents can be predicted under the Bayesian inference framework. Experiments are carried out on a challenging 8.5 hours video from a surveillance camera in the lobby of a research lab. The video contains 7 different social roles including “manager”, “researcher”, “developer”, “engineer”, “staff”, “visitor” and “mailman”. Results show that our proposed method can predict the role of each agent with high precision.
BibTeX
@inproceedings{zhang2011inferring,
  author = {Jiangen Zhang and Wenze Hu and Benjamin Z. Yao and Yongtian Wang and Song-Chun Zhu},
  title = {{Inferring social roles in long timespan video sequence}},
  booktitle = {Workshops @ ICCV},
  year = {2011},
  url = {https://openreview.net/forum?id=IpnZ3sPD84},
}
Figure