A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation

Chenliang Xu, Richard F. Doell, Stephen José Hanson, Catherine Hanson, Jason J. Corso

Abstract

Existing methods in the semantic computer vision community seem unable to deal with the explosion and richness of modern, open-source and social video content. Although sophisticated methods such as object detection or bag-of-words models have been well studied, they typically operate on low level features and ultimately suffer from either scalability issues or a lack of semantic meaning. On the other hand, video supervoxel segmentation has recently been established and applied to large scale data processing, which potentially serves as an intermediate representation to high level video semantic extraction. The supervoxels are rich decompositions of the video content: they capture object shape and motion well. However, it is not yet known if the supervoxel segmentation retains the semantics of the underlying video content. In this paper, we conduct a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation. Our study has human observers watching supervoxel segmentation videos and trying to discriminate both actor (human or animal) and action (one of eight everyday actions). We gather and analyze a large set of 640 human perceptions over 96 videos in 3 different supervoxel scales. Furthermore, we conduct machine recognition experiments on a feature defined on supervoxel segmentation, called supervoxel shape context, which is inspired by the higher order processes in human perception. Our ultimate findings suggest that a significant amount of semantics have been well retained in the video supervoxel segmentation and can be used for further video analysis.

Introduction

In computer vision, a significant "semantic gap" exists between low-level features, such as raw pixels, and high-level concepts, such as identifying a specific action. Video supervoxel segmentation—grouping pixels into consistent spatiotemporal volumes—has emerged as a popular intermediate representation. However, a critical question remained: does this abstraction process preserve the underlying meaning of the video? This study investigates whether humans and machines can still recognize actors and actions when the original texture and color are replaced by abstract supervoxel segments.

Methodology

The researchers employed a two-pronged approach to measure semantic retention:

  1. Human Perception Study: Participants were shown videos processed into a hierarchy of supervoxels at three scales: Fine, Medium, and Coarse. These stimuli removed original visual data, leaving only randomly colored spatiotemporal segments. Twenty observers were tasked with identifying the actor type and action category as quickly as possible.
  2. Machine Recognition: To benchmark automated systems, the authors developed Supervoxel Shape Context (SSC). This feature builds log-polar histograms of supervoxel edges relative to the center of mass of the video's optical flow, attempting to mimic the shape-based recognition observed in humans.
Supervoxel segmentation at Fine, Medium, and Coarse scales, illustrating the abstraction of visual data.

Dataset and Stimuli

The study utilized a curated dataset of 32 "wild" YouTube videos. These videos featured two actor types (Human and Animal) performing eight distinct actions: climbing, crawling, eating, flying, jumping, running, spinning, and walking. The dataset was further balanced between static and moving/complex backgrounds to test the robustness of recognition under varying environmental conditions.

The dataset includes eight action categories performed by humans and animals across static and moving backgrounds.

Key Insights and Results

The Coarse Advantage

Counter-intuitively, humans performed better on "Coarse" segmentations than "Fine" ones. In fine-grained versions, the high density of segments creates visual noise that obscures the subject. Coarser segments tend to merge the actor's parts into a more recognizable, holistic shape, facilitating faster and more accurate recognition.

Human vs. Machine Performance

Humans achieved an average accuracy of 82% for actor discrimination and 70.4% for action recognition across all scales. Humans were notably better at recognizing human actors (75%) compared to animals (65.9%), suggesting a biological tuning to human skeletal motion.

In contrast, machines struggled with the temporal complexity of actions. While the proposed SSC feature outperformed baselines like Dense Trajectories (DT) and Action Bank in actor recognition at the coarse level (54.52%), it lagged significantly in action recognition. This indicates that while supervoxels successfully capture shape, current machine models do not yet interpret the temporal "flow" of these shapes as effectively as the human visual system.

Conclusion

This study confirms that a significant amount of semantic information is retained within video supervoxel segmentations. The findings highlight that motion serves as a primary locator for actors, while the resulting spatiotemporal shapes provide the necessary cues for action identification. Future work will focus on developing more sophisticated machine learning models that can better leverage the rich spatiotemporal boundaries provided by supervoxels to close the performance gap between human and machine perception.

Summary updated

Abstract

Existing methods in the semantic computer vision community seem unable to deal with the explosion and richness of modern, open-source and social video content. Although sophisticated methods such as object detection or bag-of-words models have been well studied, they typically operate on low level features and ultimately suffer from either scalability issues or a lack of semantic meaning. On the other hand, video supervoxel segmentation has recently been established and applied to large scale data processing, which potentially serves as an intermediate representation to high level video semantic extraction. The supervoxels are rich decompositions of the video content: they capture object shape and motion well. However, it is not yet known if the supervoxel segmentation retains the semantics of the underlying video content. In this paper, we conduct a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation. Our study has human observers watching supervoxel segmentation videos and trying to discriminate both actor (human or animal) and action (one of eight everyday actions). We gather and analyze a large set of 640 human perceptions over 96 videos in 3 different supervoxel scales. Furthermore, we conduct machine recognition experiments on a feature defined on supervoxel segmentation, called supervoxel shape context, which is inspired by the higher order processes in human perception. Our ultimate findings suggest that a significant amount of semantics have been well retained in the video supervoxel segmentation and can be used for further video analysis.
BibTeX
@article{xu2013study,
  author = {Chenliang Xu and Richard F. Doell and Stephen José Hanson and Catherine Hanson and Jason J. Corso},
  title = {{A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation}},
  journal = {Int. J. Semantic Comput.},
  year = {2013},
  url = {https://openreview.net/forum?id=gn4a5W7HhHK5},
}
Figure