Abstract
A Model-based Approach for Combined Tracking and Resolution Enhancement of Faces in Low Resolution Video
Introduction
Wide-area surveillance often produces low-resolution (LR) video where faces consist of only a few pixels, making identification and recognition tasks nearly impossible. To address this, researchers employ Super-Resolution (SR) techniques to estimate a high-resolution (HR) image from multiple LR frames. However, traditional SR algorithms often assume objects are flat and rigid. Faces, being complex 3D shapes that deform through expressions, violate these assumptions. Furthermore, standard methods typically rely on initial interpolation steps that introduce artificial noise, blurring the final result.
Methodology: The Model-Based Framework
The authors propose a novel framework that integrates tracking and super-resolution into a single loop by shifting the process from 2D pixels to a 3D model domain.
Dual-Mode Tracking
The system uses a deformable 3D face model (CANDIDE-3). To maintain alignment, it employs a dual-mode tracker that combines appearance-based and geometric-based techniques. Appearance-based tracking compares the current frame to the initial frame for precision in small movements, while geometric-based tracking uses template matching between consecutive frames to handle larger displacements.
The mesh is subdivided until each polygon (quad) is smaller than a single pixel in the source video, allowing for sub-pixel registration accuracy. Non-rigidity is handled by sampling expression parameters (e.g., mouth movement) to minimize tracking error.
Model-Level Super-Resolution
Instead of upscaling the image first, the system projects the 3D mesh onto each video frame. As the face is tracked, each tiny quad on the mesh "collects" color values. The final super-resolved texture is the average of these values across multiple frames. This approach effectively "inverts" the image formation process, avoiding the noise-inducing interpolation steps common in traditional methods. The quality of the reconstruction is further refined by a threshold (ε) that discards frames with high tracking errors.
Results and Key Insights
The approach was evaluated using controlled lab videos, real-world surveillance footage, and synthetic objects. Key findings include:
- Robustness at Low Resolution: The tracker successfully followed faces as small as 28x20 pixels through various rotations and expressions.
- Superior Clarity: The model-based SR produced significantly sharper facial features (eyes, mouth) compared to standard bilinear interpolation or optical flow-based methods.
- Quantitative Improvement: Adding expression tracking improved the Root Mean Square Error (RMSE) of the tracking by up to 5.5% for higher-resolution faces, though the benefit decreased as resolution dropped.
- The Interpolation Trap: The study confirmed that traditional "upscaling" before processing introduces random noise. By using a sub-pixel 3D mesh, the same resolution boost is achieved without this degradation.
- Optimal Frame Count: Researchers found a "sweet spot" of 20–30 frames for SR. Beyond this, the risk of tracking errors outweighs the benefits of additional data, leading to increased blur.
Conclusion
By leveraging 3D geometry, this framework provides a robust solution for enhancing facial resolution in challenging surveillance conditions. Future work will focus on automating the initial mesh fitting and using these super-resolved 3D models to improve the training of face recognition classifiers under varying poses and lighting.
Summary updated