Abstract
We address the problem of learning a joint model of actors and actions in movies using weak supervision provided by scripts. Specifically, we extract actor/action pairs from the script and use them as constraints in a discriminative clustering framework. The corresponding optimization problem is formulated as a quadratic program under linear constraints. People in video are represented by automatically extracted and tracked faces together with corresponding motion features. First, we apply the proposed framework to the task of learning names of characters in the movie and demonstrate significant improvements over previous methods used for this task. Second, we explore the joint actor/action constraint and show its advantage for weakly supervised action learning. We validate our method in the challenging setting of localizing and recognizing characters and their actions in feature length movies Casablanca and American Beauty.
Introduction: The Challenge of Weak Supervision
Computer vision models typically require vast amounts of manually labeled data to recognize individuals or specific actions. In the context of feature-length films, manually annotating every frame is prohibitively expensive. Movie scripts offer a "free" source of descriptive information, but they provide only "weak supervision." A script might indicate that a character "sits down," but it lacks precise bounding boxes or timestamps. Furthermore, scripts and visual actions are often temporally misaligned—dialogue might occur while a character walks, even if the walking was described several lines earlier. This research addresses the ambiguity between textual script descriptions and visual evidence in video.
Methodology: Joint Discriminative Clustering
The core contribution is a joint discriminative clustering framework that solves name identification and action recognition simultaneously rather than as isolated tasks.
- Textual Extraction: Using a semantic role labeling parser (SEMAFOR), the system extracts (actor, action) pairs from scripts, such as
(Rick, sits down). - Visual Representation: The system automatically extracts "person tracks." Faces are represented using SIFT descriptors, while actions are captured via "dense trajectories" that track local motion.
- Optimization: The problem is formulated as a quadratic program where labels are treated as latent variables. Constraints derived from the script ensure that if a scene mentions "Rick sits down," at least one person track in that scene is labeled as Rick performing the "sit down" action.
- Handling Noise: To account for script imprecision (e.g., an action occurring off-screen), the model incorporates slack variables, allowing it to bypass script constraints when visual evidence is contradictory.
Experimental Results
The model was evaluated on two distinct feature-length movies: the black-and-white classic Casablanca and the modern color film American Beauty.
Character Naming
The proposed method demonstrated a significant leap in naming accuracy. In Casablanca, the system achieved an Average Precision (AP) of 0.73, notably outperforming previous state-of-the-art methods by Sivic et al. and Cour et al., which both scored approximately 0.63.
The "Jointness" Advantage
A key finding is that actors and actions are mutually informative. By leveraging the script constraint (Actor, Action), the model uses high-confidence actor identification to resolve ambiguous actions. For instance, in Casablanca, the AP for the "walking" action improved from 0.20 (without names) to 0.39 when joint modeling was applied.
Conclusion and Future Directions
While the system successfully identifies even infrequent side characters, it remains sensitive to the quality of automated person tracking. Failures in face tracking—often caused by characters turning away during actions like sitting—can lead to missing data. Additionally, the current action vocabulary is limited to frequent movements. Future work aims to scale this approach across multiple movies to build a more comprehensive universal vocabulary of human actions.
Summary updated