Can you tell a face from a HEVC bitstream?

Saeed Ranjbar Alvar, Hyomin Choi, Ivan V. Bajic

Abstract: Image and video analytics are being increasingly used on a massive scale. Not only is the amount of data growing, but the complexity of the data processing pipelines is also increasing, thereby exacerbating the problem. It is becoming increasingly important to save computational resources wherever possible. We focus on one of the poster problems of visual analytics -- face detection -- and approach the issue of reducing the computation by asking: Is it possible to detect a face without full image reconstruction from the High Efficiency Video Coding (HEVC) bitstream? We demonstrate that this is indeed possible, with accuracy comparable to conventional face detection, by training a Convolutional Neural Network on the output of the HEVC entropy decoder.

Introduction: The "Decoding Tax"

In the era of "Big Video," the computational cost of visual analytics is a massive bottleneck. Standard pipelines follow a rigid sequence: receive a compressed bitstream, fully decode it into pixels, and then run a heavy deep-learning model to detect objects or faces.

In the High Efficiency Video Coding (HEVC) standard, full reconstruction involves dequantization, inverse transforms, and spatial prediction—steps that consume the majority of a decoder's clock cycles. This research asks: Do we actually need pixels to see a face? By detecting features directly in the compressed domain, we can bypass the "decoding tax" and save significant energy and time in large-scale surveillance or cloud-based video processing.

Proposed Method: Compressed-Domain Detection

The authors propose a pipeline that taps into the HEVC entropy decoder output instead of waiting for full image reconstruction.

Comparison of (a) traditional pixel-domain face detection and (b) the proposed compressed-domain approach.

They extract three specific syntax elements from intra-coded (I-frame) blocks to create a 3-channel "feature image":

  1. Intra Prediction Mode (IPM): Captures how the encoder predicted block textures.
  2. Prediction Unit Size (PUS): Reflects local detail; smaller blocks often correspond to complex regions like facial features.
  3. Bin Number (BN): Serves as a proxy for bit consumption and local complexity.
Generation of a 3-channel feature image from HEVC syntax elements (IPM, PUS, and BN).

To a human, these feature images look like abstract, blocky heatmaps. However, for a Convolutional Neural Network (CNN), they contain sufficient structural "fingerprints" to identify a face. The researchers trained a shallow CNN (two convolutional layers and two fully connected layers) to classify these images, resulting in a model significantly lighter than typical pixel-domain architectures.

Experimental Setup

The model was trained and tested using two benchmark datasets:

  • Labeled Faces in the Wild (LFW): 13,000 images for the "face" class.
  • ImageNet: 120,000 images for the "non-face" class.

A 9:1 ratio of non-faces to faces was used to simulate real-world conditions where faces are relatively sparse compared to background clutter.

Key Results and Insights

  • Comparable Accuracy: For moderately compressed video (QP=32), the method achieved an F1-measure of 0.99 on 128x128 patches, matching the performance of benchmark CNNs running on full pixels.
  • Significant Speedup: The shallow CNN was 2 to 3 times faster than the pixel-domain benchmark. Evaluation time per patch dropped from $82 \times 10^{-4}$ seconds to $35 \times 10^{-4}$ seconds.
  • Computational Efficiency: Entropy decoding accounts for only ~37% of total decoding time. By stopping there, the system avoids the remaining 63% of the computational work required for full reconstruction.
  • The Encoder as a Feature Extractor: The core insight is that a video encoder acts as a pre-processor. Its decisions on block size and prediction modes implicitly react to scene geometry, leaving a distinct "encoding signature" for faces.

Limitations and Future Work

While promising, several challenges remain:

  • Temporal Features: The current study focuses on I-frames. Future work will extend this to P-frames and B-frames by incorporating motion vectors.
  • Quantization Sensitivity: Performance varies with the Quantization Parameter (QP). Higher compression (higher QP) reduces detail, making detection harder.
  • From Classification to Detection: The current system classifies patches. The next step is evolving it into a full detector capable of precisely localizing multiple faces in high-resolution frames.

Summary updated

Abstract

Image and video analytics are being increasingly used on a massive scale. Not only is the amount of data growing, but the complexity of the data processing pipelines is also increasing, thereby exacerbating the problem. It is becoming increasingly important to save computational resources wherever possible. We focus on one of the poster problems of visual analytics -- face detection -- and approach the issue of reducing the computation by asking: Is it possible to detect a face without full image reconstruction from the High Efficiency Video Coding (HEVC) bitstream? We demonstrate that this is indeed possible, with accuracy comparable to conventional face detection, by training a Convolutional Neural Network on the output of the HEVC entropy decoder.
BibTeX
@inproceedings{alvar2018can,
  author = {Saeed Ranjbar Alvar and Hyomin Choi and Ivan V. Bajic},
  title = {{Can you tell a face from a HEVC bitstream?}},
  booktitle = {MIPR},
  year = {2018},
  url = {https://openreview.net/forum?id=hxNAp0eqm3},
}
Figure