An Interactive Annotation Tool for Perceptual Video Compression

Evgenya Pergament, Pulkit Tandon, Kedar Tatwawadi, Oren Rippel, Lubomir Bourdev, Bruno A. Olshausen, Tsachy Weissman, Sachin Katti, Alexander G. Anderson

Abstract

Human perception is at the core of lossy video compression and yet, it is challenging to collect data that is sufficiently dense to drive compression. In perceptual quality assessment, human feedback is typically collected as a single scalar quality score indicating preference of one distorted video over another. In reality, some videos may be better in some parts but not in others. We propose an approach to collecting finer-grained feedback by asking users to use an interactive tool to directly optimize for perceptual quality given a fixed bitrate. To this end, we built a novel web-tool which allows users to paint these spatio-temporal importance maps over videos. The tool allows for interactive successive refinement: we iteratively re-encode the original video according to the painted importance maps, while maintaining the same bitrate, thus allowing the user to visually see the trade-off of assigning higher importance to one spatio-temporal part of the video at the cost of others. We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study - and are 1.9 times more likely to be preferred by viewers.

Introduction

Video content accounts for over 80% of global internet traffic, placing immense pressure on the efficiency of compression algorithms. Traditionally, these algorithms optimize for mathematical metrics such as Peak Signal-to-Noise Ratio (PSNR). However, PSNR often fails to align with human visual perception. While researchers have attempted to use automated "saliency maps" to predict human attention, these models frequently struggle to account for how the presence of compression artifacts themselves can shift a viewer's focus.

To bridge this gap, a collaborative team from Stanford, WaveOne, and UC Berkeley has developed a more direct solution: an interactive system that allows humans to manually define and refine importance maps, putting the human observer directly into the optimization loop.

The Interactive Annotation Tool

The core of this research is a novel web-based tool designed for "Successive Refinement." Unlike traditional quality assessment where a user provides a single score, this tool allows for dense, spatio-temporal feedback. The process follows a four-step iterative cycle:

  1. Annotation: Users are presented with a distorted video and an "importance map" canvas. Using a brush tool, they paint areas they wish to see in higher quality (represented in red) or erase areas deemed less important.
  2. Communication: The generated importance map is transmitted to a back-end server.
  3. Re-encoding: The server re-encodes the original video using the x264 codec. The encoder is modified to accept a $\Delta QP$-map, which assigns a lower Quantization Parameter (higher quality) to red areas and a higher QP (lower quality) to black areas, all while maintaining a strictly fixed total bitrate.
  4. Feedback: The re-encoded video is returned to the user. Because the bitrate is fixed, the user immediately perceives the trade-offs—for example, clarifying a subject's face may introduce noise in the background.
Overview of the interactive annotation system and the web-based tool interface.

To minimize the manual labor involved in annotating every frame, the tool incorporates optical flow (via the Farneback algorithm) to automatically propagate user annotations to subsequent frames. Users can then jump to specific frames to make manual corrections where the automated propagation fails.

Experimental Setup and Validation

The effectiveness of the tool was evaluated using 10 short video clips (3-5 seconds) from the CLIC 2021 dataset, covering diverse categories such as Sports, Gaming, Vlogs, and Animation. Initial importance maps were collected from 17 "master" workers on Amazon Mechanical Turk.

To validate the results, a separate group of 26 subjects participated in a two-alternative forced choice (2AFC) study. They were asked to compare the human-optimized videos against standard x264 encodes at the identical bitrate.

Key Results and Visual Impact

The study found that videos encoded using human-generated importance maps were 1.9 times more likely to be preferred over the baseline x264 encoding. The method showed particular strength in categories with clear subjects of interest, such as "Vlog" and "Sports," outperforming the baseline in 7 out of 10 tested videos.

Comparison between default x264 encoding and encoding using the human-generated importance map.

As seen in the visual comparisons, the tool allows users to make critical "perceptual budget" decisions. By sacrificing quality in static or less relevant background areas, the encoder can preserve fine details in high-importance regions, such as the facial features of a performer, leading to a significantly higher perceived quality overall.

Discussion and Future Work

While successful, the research highlighted specific challenges. In complex scenes with multiple subjects (e.g., two distinct characters in an animation), different users often focused on different areas. Averaging these "competing" maps resulted in a diluted importance map that offered little improvement over the baseline. Additionally, "over-optimization" sometimes occurred, where users focused so intensely on a central subject that the background became distractingly distorted.

The ultimate goal of this work is not to require manual annotation for all video content, but to use the dense, high-quality datasets generated by this tool to train neural networks. These networks could eventually predict human-centric importance maps automatically, leading to a new generation of perceptually-aware video compression standards.

Summary updated

Abstract

Human perception is at the core of lossy video compression and yet, it is challenging to collect data that is sufficiently dense to drive compression. In perceptual quality assessment, human feedback is typically collected as a single scalar quality score indicating preference of one distorted video over another. In reality, some videos may be better in some parts but not in others. We propose an approach to collecting finer-grained feedback by asking users to use an interactive tool to directly optimize for perceptual quality given a fixed bitrate. To this end, we built a novel web-tool which allows users to paint these spatio-temporal importance maps over videos. The tool allows for interactive successive refinement: we iteratively re-encode the original video according to the painted importance maps, while maintaining the same bitrate, thus allowing the user to visually see the trade-off of assigning higher importance to one spatio-temporal part of the video at the cost of others. We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study - and are 1.9 times more likely to be preferred by viewers. The code for the tool and dataset can be found at https://github.com/jenyap/video-annotation-tool.git
BibTeX
@inproceedings{pergament2022interactive,
  author = {Evgenya Pergament and Pulkit Tandon and Kedar Tatwawadi and Oren Rippel and Lubomir Bourdev and Bruno A. Olshausen and Tsachy Weissman and Sachin Katti and Alexander G. Anderson},
  title = {{An Interactive Annotation Tool for Perceptual Video Compression}},
  booktitle = {QoMEX},
  year = {2022},
  url = {https://openreview.net/forum?id=5vn2DKbuN7w},
}
Figure