An Interactive Annotation Tool for Perceptual Video Compression

Evgenya Pergament, Pulkit Tandon, Kedar Tatwawadi, Oren Rippel, Lubomir D. Bourdev, Bruno A. Olshausen, Tsachy Weissman, Sachin Katti, Alexander G. Anderson

Painting for Better Pixels: An Interactive Approach to Video Compression

Video accounts for over 80% of internet traffic, making efficient compression a holy grail for engineers. But here is the catch: most compression algorithms optimize for mathematical metrics like PSNR (Peak Signal-to-Noise Ratio) rather than how humans actually see. While we’ve tried using "saliency maps" to predict what people look at, these models often fail to account for how compression artifacts themselves shift our attention.

A team from Stanford, WaveOne, and UC Berkeley has proposed a more direct solution: let the humans paint the importance maps themselves.

TL;DR

Researchers developed a web-based tool that allows users to "paint" spatio-temporal importance maps over a video. By iteratively re-encoding the video at a fixed bitrate based on these maps, users can directly optimize perceptual quality, resulting in videos that are 1.9 times more likely to be preferred by viewers than standard encodes.

What Problem Does This Paper Address?

Traditional perceptual quality assessment is "low-resolution." Usually, a human viewer watches two videos and gives a single scalar score (e.g., "Video A is a 4/5"). This tells an engineer that a video is "bad," but it doesn't explain why or where. Was it the blocky artifacts on the actor's face? Or the blurring of the background text?

Existing automated "saliency" models try to guess what we find important, but they have a "chicken and egg" problem: human attention changes based on the compression artifacts present. To truly optimize for the human eye, we need a way to collect dense, spatio-temporal feedback that accounts for the trade-offs inherent in lossy compression.

Methods

The authors built a novel interactive web tool that puts the human in the loop of the compression process. The workflow follows a four-step "Successive Refinement" cycle:

  1. Annotation: The user is shown a distorted video and an "importance map" canvas. They use a brush tool to paint areas they want to see in higher quality (red) or erase areas that are less important (black).
  2. Communication: The importance map is sent to a back-end server.
  3. Re-encoding: The server re-encodes the video using x264. Crucially, it uses a modified version of the encoder that accepts a $\Delta QP$-map. This map tells the encoder to use a lower Quantization Parameter (higher quality) in the red areas and a higher QP (lower quality) in the black areas, all while maintaining a fixed total bitrate.
  4. Feedback: The newly encoded video is sent back to the user. Because the bitrate is fixed, the user immediately sees the trade-off: making the face clearer might make the background noisier.

To make the task less tedious, the tool uses optical flow (the Farneback algorithm) to propagate the user's "paint" to future frames automatically, though users can jump to any frame to make manual corrections.

Datasets

The researchers evaluated their tool using:

  • CLIC 2021 Dataset: 10 short videos (3-5 seconds) across categories like Sports, Gaming, Vlog, and Animation.
  • User Data: 17 "master" workers from Amazon Mechanical Turk were used to collect the initial importance maps.
  • Validation Study: A separate group of 26 subjects participated in a "two-alternative forced choice" (2AFC) study to compare the human-optimized videos against standard x264 encodes.

Key Insights and Ideas

  • The Power of the Loop: By showing users the immediate consequences of their annotations at a fixed bitrate, the tool forces them to make "perceptual budget" decisions. This results in data that is much more "compression-aware" than static saliency maps.
  • Normalization is Necessary: To prevent users from simply painting the whole frame red (which would defeat the purpose of a fixed bitrate), the system normalizes the importance map to a constant average value.
  • Human Consensus (and lack thereof): For many videos (like "Sports" or "Vlog"), users generally agreed on what was important (the athlete, the speaker). However, in complex scenes, "competing" interests between different annotators can actually degrade the final average map.

Key Results

  • Significant Preference: In the subjective validation study, videos encoded with human-generated importance maps were 1.9 times more likely to be preferred over the baseline x264 encoding at the same bitrate.
  • Performance by Category: The method outperformed the baseline on 7 out of 10 videos. It performed particularly well in "Vlog" and "Sports" categories where the subject of interest is clear.
  • Quantifiable Improvement: Even with the "budget" taken away from the background, the sharper reconstructions in high-importance areas led to a much higher perceived quality.

Limitations and Future Work

The paper identifies two fascinating "failure cases":

  1. The "Agnostic" Map: In an animation with two distinct characters, different users focused on different ones. Averaging these maps resulted in a "diluted" map that didn't help either character much, leading to a video that performed worse than the baseline.
  2. Over-optimization: In a gaming video, users focused so heavily on the central character that the background became distractingly distorted. This suggests that future versions of the tool might need "safety rails" to prevent the background quality from dropping below a certain threshold.

The authors suggest that the ultimate goal isn't to have humans paint every video on the internet, but to use this high-quality, dense dataset to train neural networks that can predict these human-centric importance maps automatically.

Source: An Interactive Annotation Tool for Perceptual Video Compression (2022)

Summary updated

Abstract

Human perception is at the core of lossy video compression and yet, it is challenging to collect data that is sufficiently dense to drive compression. In perceptual quality assessment, human feedback is typically collected as a single scalar quality score indicating preference of one distorted video over another. In reality, some videos may be better in some parts but not in others. We propose an approach for collecting finer-grained user feedback through an interactive tool that allows direct optimization of perceptual quality given a fixed bitrate. To this end, we built a novel web-tool which allows users to paint spatio-temporal importance maps over videos. The tool allows for interactive successive refinement: we iteratively re-encode the original video according to the painted importance maps, while maintaining the same bitrate, thus allowing the user to visually see the trade-off of assigning higher importance to one spatio-temporal part of the video at the cost of others. We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study (n = 26) - and are 1.9 times more likely to be preferred by viewers. We plan on collecting a large-scale dataset using the tool for automated perceptual compression in the future. The code for the tool and dataset can be found at https://github.com/jenyap/video-annotation-tool.git.
BibTeX
@inproceedings{pergament2022interactive,
  author = {Evgenya Pergament and Pulkit Tandon and Kedar Tatwawadi and Oren Rippel and Lubomir D. Bourdev and Bruno A. Olshausen and Tsachy Weissman and Sachin Katti and Alexander G. Anderson},
  title = {{An Interactive Annotation Tool for Perceptual Video Compression}},
  booktitle = {QoMEX},
  year = {2022},
  url = {https://openreview.net/forum?id=5vn2DKbuN7w},
}

Loading related content...

Figure