Online Lens Motion Smoothing for Video Autofocus

Abdullah Abuolaim, Michael BrownWACV 2020

TL;DR

This paper tackles the problem of "lens wobble" in video autofocus (AF) by proposing two novel online smoothing methods: a supervised BLSTM and an unsupervised WMA. They demonstrate that users strongly prefer videos with smoother lens motion, even at the cost of a slight decrease in sharpness, and their methods achieve significant lens motion reduction with minimal sharpness loss.

What Problem Does This Paper Address?

Autofocus (AF) systems in cameras are designed to ensure that the desired scene content is sharp. While this is straightforward for single images, applying the same logic to video capture leads to a common artifact known as "lens wobble." This occurs because conventional video AF algorithms optimize sharpness for each individual frame, often resulting in rapid and jerky lens movements. These movements are perceptible to viewers and can be distracting, detracting from the overall viewing experience. The challenge is to smooth these lens movements without sacrificing too much image sharpness, and crucially, to do this in real-time (online) as the video is being captured.

Methods

The authors propose two novel online methods for smoothing lens motion in video AF:

  • Supervised Bidirectional Long Short-Term Memory (BLSTM): This method uses a deep learning approach. They first generate "ground truth" smooth lens trajectories by applying an offline smoothing filter (Savitzky-Golay) to conventional AF outputs. This offline smoothed data is then used to train a BLSTM network. The BLSTM takes the current lens position and the optimal lens target (from the camera's PDAF system) as input and predicts the next lens movement (forward, backward, or no movement). The bidirectional nature allows it to consider both past and future (in the training data) context for smoother predictions.
  • Unsupervised Weighted Moving Average (WMA): This is a simpler, computationally less intensive method that doesn't require offline training data. It uses a weighted moving average of past lens positions. The key innovation here is that it assigns weights based on the difference between the current lens position and the target optimal lens position. This encourages larger movements when the target is far off and suppresses movements when the target is close, effectively smoothing the trajectory without explicit training.

Datasets

The primary dataset used is the "4D temporal focal stack dataset and AF software platform" introduced by Abuolaim et al. [1]. This dataset consists of ten scenes, each with multiple focal stacks (50 images per stack), simulating a camera's AF system. They generated 36 videos for their experiments, covering three main AF objectives (global, 9-focus-points, and 51-focus-points) and a face region objective.

Key Insights and Ideas

  • User Preference for Smoothness: The most significant insight is that users strongly prefer videos with smoother lens motion, even if it means a slight compromise in peak sharpness. This directly challenges the conventional wisdom of maximizing sharpness at all costs for video.
  • Online Smoothing is Crucial: Offline smoothing methods (like SG) produce excellent results but are not feasible for real-time AF systems because they require knowledge of future frames. The paper's focus on online methods is therefore critical for practical applications.
  • WMA's Simplicity and Effectiveness: The unsupervised WMA method achieves comparable or even slightly better results in terms of sharpness change compared to the more complex BLSTM, while being much simpler to implement and computationally cheaper. This makes it a very attractive option for existing hardware.
  • Face Region Exception: While generally users prefer smoothness, there's an exception for face regions, where users will tolerate larger lens movements to ensure the face is in focus.

Key Results

  • Lens Motion Reduction: Both BLSTM and WMA significantly reduced lens movements, achieving up to 64% reduction compared to conventional AF.
  • Sharpness Change: The reduction in sharpness was minimal, with less than a 5.2% change observed. The WMA method showed a slightly smaller sharpness change than BLSTM.
  • User Preference: In user studies, both BLSTM and WMA were significantly preferred over conventional AF. The WMA method was slightly more preferred than BLSTM, indicated by smaller confidence intervals.
  • Accuracy (BLSTM): The BLSTM model achieved an average accuracy of 91.6% in predicting offline smoothed lens motion, with particularly high accuracy for the global objective.

Limitations and Future Work

  • Dataset Limitations: The dataset used has only ten scenes and relatively low temporal sampling, which could limit the generalizability of the models to real-world scenarios with more complex motion.
  • Future Work: The authors suggest that camera manufacturers providing more low-level hardware access to AF features would enable further research in this area. They also highlight the potential for WMA to be easily integrated into existing AF hardware.

Summary updated

BibTeX
@inproceedings{abuolaim2020online,
  author = {Abdullah Abuolaim and Michael Brown},
  title = {{Online Lens Motion Smoothing for Video Autofocus}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure