Semantic Question-Answering with Video and Eye-Tracking Data: AI Foundations for Human Visual Perception Driven Cognitive Film Studies

Jakob Suchan, Mehul BhattIJCAI 2016

Introduction

In visual perception research, eye-tracking typically yields "flat" statistical data such as fixation counts and heatmaps. While these metrics provide a quantitative overview, they often fail to capture the underlying meaning of a viewer's gaze. For instance, a standard heatmap cannot distinguish whether a viewer is tracking a specific protagonist across a cut or reacting to a complex camera movement like a dolly zoom. This "semantic gap" between raw pixel coordinates and high-level filmic concepts currently necessitates laborious manual annotation. This paper introduces a computational framework that bridges this gap by integrating computer vision with logic-based reasoning to automate the semantic interpretation of video and eye-tracking data.

Proposed Framework

The authors propose an integrated pipeline consisting of three functional layers designed to move from low-level sensor data to high-level cognitive theory.

The integrated pipeline for semantic visual perception analysis, showing the flow from raw data to qualitative analytics.
  1. Visual Processing: The system employs computer vision techniques to extract scene features, including optical flow for camera movement estimation and Convolutional Neural Networks (CNNs) for character identification.
  2. Space-Motion Histories: Detected objects are abstracted into "Space-Time Histories." The framework utilizes Region Connection Calculus (RCC8) for spatial relations and Allen’s Interval Algebra for temporal relations, allowing the system to determine if a gaze followed a character during a specific camera maneuver.
  3. Semantic Question-Answering: Extracted data is processed within a Constraint Logic Programming (PROLOG) environment. This allows researchers to define declarative rules—such as attn_following(Viewer, Character, Time)—and query the system for high-level insights, such as identifying when an audience followed a specific character after a cinematic cut.

Experimental Setup

The framework was evaluated using a multi-modal dataset tailored for Cognitive Film Studies. The study involved 31 participants whose eye movements were recorded at 60 Hz using a Tobii X2-60 tracker. The visual stimuli consisted of 16 scenes from 12 films, including works by Wes Anderson (e.g., The Grand Budapest Hotel and The Darjeeling Limited), chosen for their distinct and highly structured cinematographic styles.

Results and Insights

The system demonstrates the ability to translate messy, noisy gaze data into a symbolic language aligned with film grammar and narrative structure.

Analysis of cinematographic effects and character tracking in 'The Grand Budapest Hotel', illustrating how the framework identifies cuts, tilts, and viewer attention.

Key results include:

  • Character Identification: The system successfully correlated gaze points with specific characters identified by CNNs, even in complex multi-actor scenes.
  • Cinematographic Impact: The framework identified how specific devices, such as cuts or tilts, triggered shifts in aggregate spectator attention.
  • Automated Summarization: By understanding scenes symbolically, the system can generate natural language summaries of viewer behavior, such as noting when attention shifts immediately following a specific camera movement.

Conclusion

By using logic as a bridge, this research transforms eye-tracking from a purely statistical tool into a queryable semantic system. While the framework currently relies on the robustness of underlying computer vision tools, it establishes a foundation for "Cognitive Vision" with applications extending into robotics and human-centered assistive technologies where AI must understand the intent behind human visual attention.

Summary updated

BibTeX
@inproceedings{suchan2016semantic,
  author = {Jakob Suchan and Mehul Bhatt},
  title = {{Semantic Question-Answering with Video and Eye-Tracking Data: AI Foundations for Human Visual Perception Driven Cognitive Film Studies}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}

Loading related content...

Figure