Real-Time Selfie Video Stabilization

Jiyang Yu, Ravi Ramamoorthi, Ke-Li Cheng, Michel Sarkis, Ning Bi

Abstract

We propose a novel real-time selfie video stabilization method that is completely automatic and runs at 26 fps. Our approach uses a 1D linear convolutional network to directly infer rigid moving least squares (MLS) warping, balancing global rigidity with local flexibility. The network is specifically designed to stabilize both background and foreground simultaneously, offering users optional control over stabilization focus. To train the model, we collected a dataset of 1005 selfie videos, significantly larger than previous efforts. Additionally, we introduce a grid approximation method for MLS warping to enable real-time performance. Our method produces visually and quantitatively superior results compared to existing real-time general stabilization techniques and achieves comparable quality to offline methods at much higher speeds.

The Challenge of Selfie Stabilization

Stabilizing "selfie" videos presents unique difficulties compared to standard landscape footage. In a typical selfie, the foreground subject (the person) and the background often move independently and at different scales. Standard algorithms frequently fail in two ways:

  1. Foreground/Background Conflict: Stabilizing only the face causes the background to shake violently, while stabilizing only the background can make the face appear distorted or "floaty."
  2. Real-Time Constraints: High-quality stabilization typically requires computationally expensive optimizations or "look-ahead" frames, making them too slow for mobile devices. While some real-time methods exist, they often rely on hardware sensors like gyroscopes, which may be unavailable or poorly calibrated.

Real-Time Stabilization Pipeline

The proposed system addresses these challenges through a purely 2D, vision-based pipeline consisting of three main stages:

  1. Motion Detection: A segmentation network separates the subject from the background. Background motion is tracked using standard KLT feature points, while foreground motion is captured by fitting a 3D face mesh to the user.
  2. Two-Branch Linear Network: The core of the system is a specialized network with separate input branches for background features and face vertices. Surprisingly, the authors found that a 1D linear convolutional network (without non-linear activations like ReLU) outperformed deeper networks. This is because the relationship between input shake and required compensation is inherently linear.
  3. Grid-Approximated MLS Warping: To transform shaky frames into stable ones, the system employs Moving Least Squares (MLS) warping. This technique preserves face rigidity while allowing background flexibility. To maintain 26 fps, the complex MLS field is approximated using a sparse grid followed by dense bilinear interpolation.
The system tracks background feature points (green) and foreground face vertices (red) to drive the stabilization process.

Key Insights: Linearity and User Control

A counter-intuitive finding of this research is that "Linearity is King." Removing non-linear activation layers improved performance because stabilization is a linear problem: doubling the input shake requires doubling the output compensation. A linear network captures this relationship accurately without the artifacts common in traditional deep networks.

Furthermore, the system introduces a weighting parameter ($\lambda$), allowing users to choose their stabilization priority. A high $\lambda$ keeps the face rock-steady (ideal for vlogging), while a lower $\lambda$ prioritizes a smooth background.

Performance and Results

The system achieves a processing speed of 26 fps on a desktop GPU, representing a speedup of several orders of magnitude over previous high-quality selfie stabilization methods. In quantitative tests, it outperformed state-of-the-art real-time methods like MeshFlow and Steadiface in both stability and distortion metrics.

The final stabilized output (C) is generated using grid-approximated Moving Least Squares (MLS) warping.

Qualitatively, the method avoids the "wobbly face" syndrome. By using the face mesh as a constraint, the subject's features remain rigid and natural even when the background is heavily warped to compensate for camera shake.

Conclusion

While the system is highly effective, it relies on detectable features and may struggle in extreme lighting or featureless environments. Future work could involve video frame completion to "fill in" edges cropped during stabilization, potentially enabling wider-angle results without resolution loss.

Summary updated

Abstract

We propose a novel real-time selfie video stabilization method. Our method is completely automatic and runs at 26 fps. We use a 1D linear convolutional network to directly infer the rigid moving least squares warping which implicitly balances between the global rigidity and local flexibility. Our network structure is specifically designed to stabilize the background and foreground at the same time, while providing optional control of stabilization focus (relative importance of foreground vs. background) to the users. To train our network, we collect a selfie video dataset with 1005 videos, which is significantly larger than previous selfie video datasets. We also propose a grid approximation method to the rigid moving least squares warping that enables the real-time frame warping. Our method is fully automatic and produces visually and quantitatively better results than previous real-time general video stabilization methods. Compared to previous offline selfie video methods, our approach produces comparable quality with a speed improvement of orders of magnitude.
BibTeX
@inproceedings{yu2021real,
  author = {Jiyang Yu and Ravi Ramamoorthi and Ke-Li Cheng and Michel Sarkis and Ning Bi},
  title = {{Real-Time Selfie Video Stabilization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=cukEKvCpCG},
}

Loading related content...

Figure