Further Understanding Videos through Adverbs: A New Video Task

Bo Pang, Kaiwen Zha, Yifan Zhang, Cewu Lu

Introduction: Moving Beyond the "What" to the "How"

In the field of computer vision, researchers have achieved remarkable success in identifying nouns (object detection) and verbs (action recognition). While current models can reliably determine if a person is "running," they often fail to capture the crucial nuances of how that action is performed. In their AAAI-20 paper, researchers from Shanghai Jiao Tong University introduce a new frontier in video understanding: Behavior Adverbs (BA). This task moves beyond simple action labels to describe the mood, attitude, and style of human behavior, providing a more complete semantic picture of video content.

Examples of action-adverb pairs from the VAAD dataset, illustrating how adverbs provide essential context to actions like smoking, running, and exercising.

The Challenge of Adverb Recognition

Standard action recognition is primarily "verb-centric." However, there is a profound semantic difference between "smoking happily" and "smoking sadly," or "running professionally" versus "running in fear." Adverbs represent "deeper" concepts than actions because their recognition depends on several complex factors:

  1. Subtle Dynamics: Identifying the "strength" or "speed" of a movement requires analyzing the first and second-order derivatives of displacement, which are often overlooked by standard models.
  2. Facial Expressions: Many mood-related adverbs (e.g., sadly, happily) are primarily conveyed through facial cues rather than body movement.
  3. Contextual Relations: Relational adverbs (e.g., "in fear") often require understanding the environment or the presence of other subjects in the scene.

Methodology: The BA Understanding Network (BAUN)

To address these complexities, the authors proposed the BA Understanding Network (BAUN). Unlike standard 3D Convolutional Neural Networks (CNNs) that often conflate spatial and temporal information, BAUN employs a multi-faceted approach to capture the diverse cues necessary for adverb recognition.

The BAUN architecture integrates an Expression Model, a Spatial-Temporal Separated Module (STSM), and an Image Relation Network (IRN) to capture nuanced behavioral cues.

Key components of the architecture include:

  • Expression Model: A dedicated module for extracting facial features to identify mood-related adverbs.
  • Spatial-Temporal Separated Module (STSM): This module decouples spatial and temporal processing, allowing the model to better capture "speed" and "strength" without interference from static spatial features.
  • Image Relation Network (IRN): Utilizing 2D positional encoding, the IRN explores the relationship between the subject and the surrounding context, providing the necessary data for relational adverbs.
  • Conditional Distribution Aggregator: This component leverages the statistical priors between actions and adverbs (e.g., "smoking" is statistically more likely to be paired with "sadly" than "professionally") to refine final predictions.

The VAAD Dataset

The researchers introduced the Videos with Action and Adverb Dataset (VAAD) to support this new task. VAAD is a large-scale resource consisting of 150,000 video clips featuring 2,000 unique action-adverb pairs across 200 action categories and 51 adverb categories. Notably, VAAD is multi-labeled, allowing a single action to be described by multiple adverbs simultaneously, covering mood, attitude, and physical attributes.

Key Results and Insights

The proposed BAUN model achieved an 11% relative improvement over the popular I3D model on the VAAD dataset. Ablation studies highlighted the importance of context, as removing the IRN led to a 6.7% drop in performance.

A significant finding was the mutual reinforcement between actions and adverbs. The researchers discovered that training models to recognize adverbs improved their performance on standard action recognition tasks by an average of 3.4%. This suggests that learning the "how" provides critical features that help disambiguate the "what." Furthermore, models pre-trained on VAAD demonstrated superior transferability to other benchmarks like UCF-101 and Kinetics.

Conclusion and Future Work

While BAUN excels at identifying "physical" adverbs (e.g., slowly, professionally), recognizing "psychological" adverbs (e.g., painfully, hesitantly) remains a significant challenge, with Average Precision scores often falling below 5%. Future research will likely focus on more sophisticated relational reasoning and the integration of multi-modal cues, such as audio and long-term temporal context, to better decode the complexities of human emotion and attitude in video.

Summary updated

Abstract

Video understanding is a research hotspot of computer vision and significant progress has been made on video action recognition recently. However, the semantics information contained in actions is not rich enough to build powerful video understanding models. This paper first introduces a new video semantics: the Behavior Adverb (BA), which is a more expressive and difficult one covering subtle and inherent characteristics of human action behavior. To exhaustively decode this semantics, we construct the Videos with Action and Adverb Dataset (VAAD), which is a large-scale dataset with a semantically complete set of BAs. The dataset will be released to the public with this paper. We benchmark several representative video understanding methods (originally for action recognition) on BA and action recognition. The results show that BA recognition task is more challenging than conventional action recognition. Accordingly, we propose the BA Understanding Network (BAUN) to solve this problem and the experiments reveal that our BAUN is more suitable for BA recognition (11% better than I3D). Furthermore, we find these two semantics (action and BA) can propel each other forward to better performance: promoting action recognition results by 3.4% averagely on three standard action recognition datasets (UCF-101, HMDB-51, Kinetics).
BibTeX
@inproceedings{pang2020further,
  author = {Bo Pang and Kaiwen Zha and Yifan Zhang and Cewu Lu},
  title = {{Further Understanding Videos through Adverbs: A New Video Task}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6855},
}
Figure