The Face Object based HEVC System for Video Call

Xi Wang, Chenggang Yan, Qingming Huang, Li Su, Shuqiang Jiang, Xianglin Huang

Abstract

Standard video compression standards like HEVC often struggle with the complex motions typical of video calls, such as head rotations and scaling. This paper proposes an object-based HEVC enhancement that treats the human face as a single transforming entity. By utilizing affine transformations to align reference frames with the current face object, the system achieves significant bitrate reductions and improved visual quality. Experimental results demonstrate average bitrate savings of 10.7%, reaching up to 19.59% in specific scenarios, while maintaining high reconstruction fidelity.

Introduction

In modern video communication, the human face is the primary focus of attention. However, standard video codecs like High Efficiency Video Coding (HEVC) treat the frame as a collection of independent square blocks. This traditional approach relies on translation motion models, assuming objects move only linearly (up, down, left, or right).

During video calls, users frequently perform non-linear movements, such as tilting their heads (rotation) or moving closer to the camera (scaling). When these movements occur, block-based encoders struggle to find accurate matches in previous frames, leading to high residual errors and increased data requirements to maintain quality. This paper presents a specialized HEVC scheme designed to overcome these limitations by treating the face as a single, transforming object.

Proposed Face-Aware HEVC System

The proposed system integrates face detection and alignment directly into the HEVC encoding pipeline. The process follows four primary stages:

  1. Face Detection and Landmark Mapping: The system utilizes the STASM algorithm to locate 76 facial feature points. To define the face region efficiently, it tracks seven key points, including the eyes and face boundaries.
Facial landmark detection using 7 key points to define the face region.
  1. Affine Alignment: Instead of simple translation, the system calculates an affine transformation matrix ($\tau$) that accounts for scaling, rotation, and translation. This matrix is used to "warp" a previous reference frame to match the orientation and size of the face in the current frame.
  2. Modified Referent Picture Set (RPS): The encoder generates a new, aligned reference picture ($R'$) based on the affine transformation. This virtual reference frame is added to the RPS, allowing the standard HEVC tools to predict the current face from a perfectly aligned version of the previous one.
  3. Encoding and Transmission: The six elements of the transformation matrix are compressed and stored in the slice header of the video stream, ensuring the decoder can replicate the alignment process.
Flowchart of the proposed face-aware encoding pipeline integrating affine alignment.

Experimental Results

The researchers evaluated the system using standard sequences (Johnny, KristenAndSara, and suzie) and a custom sequence (GreenCloth). The results indicate that the affine alignment effectively handles head movement, leading to a "Motion Vector Collapse." In the GreenCloth sequence, the average motion vector magnitude dropped from 8.40 in standard HEVC to just 0.29, confirming that the global transformation captures the majority of the movement.

Key performance metrics include:

  • Bitrate Savings: An average saving of 10.7% was achieved across test cases, peaking at 19.59% for the Johnny sequence.
  • Visual Quality: The method provided a consistent increase in Peak Signal-to-Noise Ratio (PSNR), with an average improvement of 0.31 dB in high-quality settings.
  • Utilization: On average, 57.29% of the coding units within the face region selected the new aligned reference picture, validating the effectiveness of the object-based approach.

Conclusion

By shifting from local block matching to global object transformation, this face-aware HEVC system significantly improves compression efficiency for video calls. While the current implementation introduces computational overhead for face detection and matrix calculation, it demonstrates the potential for massive bandwidth savings in mobile communication. Future work will focus on optimizing the encoder's speed and improving the handling of complex occlusions.

Summary updated

BibTeX
@article{wang2015face,
  author = {Xi Wang and Chenggang Yan and Qingming Huang and Li Su and Shuqiang Jiang and Xianglin Huang},
  title = {{The Face Object based HEVC System for Video Call}},
  journal = {EAI Endorsed Trans. Future Intell. Educ. Environ.},
  year = {2015},
  doi = {10.4108/icst.mobimedia.2015.258957},
}
Figure