Movie editing and cognitive event segmentation in virtual reality video

Ana Serrano, Vincent Sitzmann, Jaime Ruiz-Borau, Gordon Wetzstein, Diego Gutierrez, Belén MasiáACM Trans. Graphics 2017 · DOI · OpenReview

Abstract

Traditional cinematography has relied for over a century on a well-established set of editing rules, called continuity editing, to create a sense of situational continuity. Despite massive changes in visual content across cuts, viewers in general experience no trouble perceiving the discontinuous flow of information as a coherent set of events. However, Virtual Reality (VR) movies are intrinsically different from traditional movies in that the viewer controls the camera orientation at all times. As a consequence, common editing techniques that rely on camera orientations, zooms, etc., cannot be used. In this paper we investigate key relevant questions to understand how well traditional movie editing carries over to VR. To do so, we rely on recent cognition studies and the event segmentation theory, which states that our brains segment continuous actions into a series of discrete, meaningful events. We first replicate one of these studies to assess whether the predictions of such theory can be applied to VR. We next gather gaze data from viewers watching VR videos containing different edits with varying parameters, and provide the first systematic analysis of viewers' behavior and the perception of continuity in VR. From this analysis we make a series of relevant findings; for instance, our data suggests that predictions from the cognitive event segmentation theory are useful guides for VR editing; that different types of edits are equally well understood in terms of continuity; and that spatial misalignments between regions of interest at the edit boundaries favor a more exploratory behavior even after viewers have fixated on a new region of interest. In addition, we propose a number of metrics to describe viewers' attentional behavior in VR. We believe the insights derived from our work can be useful as guidelines for VR content creation.

Introduction

In traditional filmmaking, the director maintains absolute control over the frame. Techniques such as the "180-degree rule" and match-on-action cuts create a sense of "invisible" editing, allowing viewers to perceive a continuous narrative despite visual discontinuities. In Virtual Reality (VR), however, the viewer assumes the role of the camera operator. This autonomy presents a significant challenge for editors: how can one transition between scenes without disorienting the viewer or breaking narrative flow?

This research bridges cognitive science and cinematography to establish systematic rules for editing in 360° VR. By applying Event Segmentation Theory (EST)—the cognitive principle that the brain naturally divides continuous experience into discrete, meaningful events—the authors investigate how immersive environments affect the perception of cinematic continuity.

Methodology

The researchers conducted two primary studies to quantify viewer behavior:

1. Event Segmentation Replication

To determine if VR users segment experiences similarly to traditional film viewers, the authors replicated a classic cognitive experiment using narrative VR content (e.g., Star Wars - Hunting of the Fallen). Participants marked "event boundaries" while watching 360° movies. The study categorized edits into three types: action discontinuities (E1), spatial/temporal discontinuities (E2), and continuity edits (E3).

Figure 2: Analysis of different edit types (E1, E2, E3) and the percentage of viewers who identified them as event boundaries.

2. Systematic Eye-Tracking Study

The authors developed a controlled dataset of 216 clips to analyze attentional behavior. They systematically varied three variables:

  • Edit Type: Comparing jump cuts (discontinuous) vs. match-on-action (continuous).
  • ROI Alignment: The angular distance between the Region of Interest (ROI) before and after a cut (0°, 40°, or 80°).
  • ROI Configuration: The number and distribution of focal points within the 360° field of view.
Figure 6: Examples of the ROI configurations used in the systematic study, including single and multiple regions of interest across different fields of view.

Key Insights and Results

Cognitive Consistency

The results confirmed that human brains process VR narratives similarly to traditional films. Action discontinuities (E1) were the strongest predictors of a perceived new event, accounting for approximately 59% of marked boundaries. In contrast, continuity edits (E3) successfully maintained flow, being marked as boundaries only 16% of the time.

The Exponential Search Penalty

A critical finding was the relationship between spatial misalignment and viewer attention. The time required for a viewer to locate a new ROI after a cut follows an exponential trend relative to the degrees of misalignment. This is modeled by the formula $f(x) = 8.8e^{0.025x}$, where $x$ represents the degrees of misalignment.

The "Search Mode" Trigger

Large spatial misalignments (specifically at 80°) do more than just delay the viewer; they trigger a distinct "search mode." This exploratory behavior, characterized by rapid eye movements (saccades), persists even after the viewer has successfully fixated on the new subject. There is a statistically significant drop in the number of stable fixations ($p=0.003$) at high misalignment levels.

Gaze Anchoring

Cuts that follow a continuous action (match-on-action) serve as "gaze anchors." These edits attract attention to the new ROI significantly faster and more consistently than jump cuts. Conversely, having multiple ROIs before a cut leads to increased visual wandering afterward, as viewers search for previously established secondary subjects.

Conclusion

This study provides the first data-driven guidelines for VR cinematography. By understanding the exponential penalty of spatial misalignment and the anchoring power of continuity edits, creators can better guide viewer attention in 360° environments. While VR offers unprecedented viewer autonomy, the fundamental cognitive processes of event segmentation remain a reliable compass for narrative construction.

Summary updated

Abstract

Traditional cinematography has relied for over a century on a well-established set of editing rules, called continuity editing, to create a sense of situational continuity. Despite massive changes in visual content across cuts, viewers in general experience no trouble perceiving the discontinuous flow of information as a coherent set of events. However, Virtual Reality (VR) movies are intrinsically different from traditional movies in that the viewer controls the camera orientation at all times. As a consequence, common editing techniques that rely on camera orientations, zooms, etc., cannot be used. In this paper we investigate key relevant questions to understand how well traditional movie editing carries over to VR, such as: Does the perception of continuity hold across edit boundaries? Under which conditions? Does viewers' observational behavior change after the cuts? To do so, we rely on recent cognition studies and the event segmentation theory, which states that our brains segment continuous actions into a series of discrete, meaningful events. We first replicate one of these studies to assess whether the predictions of such theory can be applied to VR. We next gather gaze data from viewers watching VR videos containing different edits with varying parameters, and provide the first systematic analysis of viewers' behavior and the perception of continuity in VR. From this analysis we make a series of relevant findings; for instance, our data suggests that predictions from the cognitive event segmentation theory are useful guides for VR editing; that different types of edits are equally well understood in terms of continuity; and that spatial misalignments between regions of interest at the edit boundaries favor a more exploratory behavior even after viewers have fixated on a new region of interest. In addition, we propose a number of metrics to describe viewers' attentional behavior in VR. We believe the insights derived from our work can be useful as guidelines for VR content creation.
BibTeX
@article{serrano2017movie,
  author = {Ana Serrano and Vincent Sitzmann and Jaime Ruiz-Borau and Gordon Wetzstein and Diego Gutierrez and Belén Masiá},
  title = {{Movie editing and cognitive event segmentation in virtual reality video}},
  journal = {ACM Trans. Graphics},
  year = {2017},
  doi = {10.1145/3072959.3073668},
}
Figure