Introduction
In visual perception research, eye-tracking typically yields "flat" statistical data such as fixation counts and heatmaps. While these metrics provide a quantitative overview, they often fail to capture the underlying meaning of a viewer's gaze. For instance, a standard heatmap cannot distinguish whether a viewer is tracking a specific protagonist across a cut or reacting to a complex camera movement like a dolly zoom. This "semantic gap" between raw pixel coordinates and high-level filmic concepts currently necessitates laborious manual annotation. This paper introduces a computational framework that bridges this gap by integrating computer vision with logic-based reasoning to automate the semantic interpretation of video and eye-tracking data.
Proposed Framework
The authors propose an integrated pipeline consisting of three functional layers designed to move from low-level sensor data to high-level cognitive theory.
- Visual Processing: The system employs computer vision techniques to extract scene features, including optical flow for camera movement estimation and Convolutional Neural Networks (CNNs) for character identification.
- Space-Motion Histories: Detected objects are abstracted into "Space-Time Histories." The framework utilizes Region Connection Calculus (RCC8) for spatial relations and Allen’s Interval Algebra for temporal relations, allowing the system to determine if a gaze followed a character during a specific camera maneuver.
- Semantic Question-Answering: Extracted data is processed within a Constraint Logic Programming (PROLOG) environment. This allows researchers to define declarative rules—such as
attn_following(Viewer, Character, Time)—and query the system for high-level insights, such as identifying when an audience followed a specific character after a cinematic cut.
Experimental Setup
The framework was evaluated using a multi-modal dataset tailored for Cognitive Film Studies. The study involved 31 participants whose eye movements were recorded at 60 Hz using a Tobii X2-60 tracker. The visual stimuli consisted of 16 scenes from 12 films, including works by Wes Anderson (e.g., The Grand Budapest Hotel and The Darjeeling Limited), chosen for their distinct and highly structured cinematographic styles.
Results and Insights
The system demonstrates the ability to translate messy, noisy gaze data into a symbolic language aligned with film grammar and narrative structure.
Key results include:
- Character Identification: The system successfully correlated gaze points with specific characters identified by CNNs, even in complex multi-actor scenes.
- Cinematographic Impact: The framework identified how specific devices, such as cuts or tilts, triggered shifts in aggregate spectator attention.
- Automated Summarization: By understanding scenes symbolically, the system can generate natural language summaries of viewer behavior, such as noting when attention shifts immediately following a specific camera movement.
Conclusion
By using logic as a bridge, this research transforms eye-tracking from a purely statistical tool into a queryable semantic system. While the framework currently relies on the robustness of underlying computer vision tools, it establishes a foundation for "Cognitive Vision" with applications extending into robotics and human-centered assistive technologies where AI must understand the intent behind human visual attention.
Summary updated