Fourier Based Pre-Processing For Seeing Through Water

Jerin Geo James, Ajit RajwadeWACV 2020

TL;DR

This paper introduces a novel Fourier-based pre-processing technique to correct significant spatial distortions in videos captured through fluctuating water surfaces. By exploiting the spatio-temporal properties of water waves, their method efficiently interpolates optical flow from a sparse set of feature points, dramatically improving the performance of subsequent image restoration algorithms.

What Problem Does This Paper Address?

Capturing images or videos of scenes submerged in water, or viewed through a water surface from above, introduces significant spatial distortions. These distortions arise from the refraction of light at the dynamic, wavy water-air boundary. This phenomenon is particularly challenging because the water surface is not static; it constantly changes shape due to external forces like wind, leading to time-varying, non-rigid distortions. These distortions severely degrade the performance of standard computer vision tasks such as object tracking, segmentation, and recognition, which are crucial for applications like underwater surveillance or marine life monitoring. Existing methods for dealing with these distortions are often computationally expensive or limited in their ability to handle complex wave patterns.

Methods

The core of the proposed method lies in a Fourier-based approach to interpolate optical flow. The algorithm proceeds in several key steps:

  1. Salient Feature Point Tracking: A small number (around N=100) of salient feature points are detected in the first frame of the video and then tracked across subsequent frames using the KLT tracker. This yields "point-trajectories."
  2. Computing Displacement Trajectories: For each tracked point, an approximate "undistorted" initial position is estimated by averaging its positions over time. The difference between the actual tracked position and this estimated undistorted position over time forms the "displacement-trajectory."
  3. Fourier Decomposition: Each displacement-trajectory is then decomposed into its Fourier components. This step represents the temporal motion of each point as a sum of sinusoids.
  4. Fourier Phase Interpolation: The key insight is that for a single dominant wave, the phase factors of the Fourier coefficients of displacement-trajectories from different points exhibit a "shift-plane property." This means the phase varies linearly with the spatial coordinates of the points. The algorithm estimates the parameters of this plane using RANSAC, allowing it to infer the displacement-trajectories for non-tracked points.
  5. Handling Conflating Frequencies: When multiple waves are present and their Fourier supports overlap ("conflating frequencies"), the shift-plane property is violated. To address this, the algorithm identifies these conflating frequencies and uses a modified interpolation scheme (Eqn. 10) that only considers non-conflating frequencies.
  6. Motion Correction: Using the interpolated displacement-trajectories for all pixels, the algorithm applies a reverse deformation to each frame, effectively correcting the distortions.

This Fourier-based method acts as an efficient pre-processing step, significantly reducing the non-rigid distortions before applying other, more sophisticated restoration algorithms.

Datasets

The paper evaluates their method on two datasets of real video sequences:

  • Real1: Collected from a "wave-flume" with laminated posters at the bottom of a water tank. Waves were generated using paddles, resulting in distortions that could not be explained by single cyclic waves.
  • Real2: Consists of three video sequences acquired with a camera.

The authors mention that all image and video results are available in a supplemental material folder. No new benchmarks were introduced.

Key Insights and Ideas

  • Exploiting Spatio-Temporal Redundancy: The core idea is that the complex, dynamic motion of water waves has inherent spatio-temporal redundancies that can be exploited. The Fourier transform is a powerful tool for analyzing these periodic and quasi-periodic motions.
  • Shift-Plane Property: The observation that the phase of Fourier components of displacement-trajectories forms a plane for single-wave scenarios is a crucial geometric insight. This allows for efficient interpolation of motion from sparse to dense.
  • Fourier Separation Property (FSP): For multi-wave scenarios, the assumption that dominant Fourier components of different waves have disjoint supports is key. This allows for the separation of contributions from different waves.
  • Pre-processing Power: The most significant insight is that this Fourier-based method, while not a complete restoration solution on its own, acts as a highly effective pre-processing step. It dramatically simplifies the problem for subsequent restoration algorithms, leading to substantial improvements in their performance.

Key Results

  • Significant Motion Reduction: The Fourier-based pre-processing stage achieves approximately 50% motion reduction in a median sense, as quantified by the Motion Reduction (MR) metric.
  • Improved Image Quality: When used as a pre-processing step, the Fourier method significantly improves the performance of state-of-the-art restoration methods like LWB [21] and SBR [17]. This improvement is measured by higher Normalized Mutual Information (NMI) and Structural Similarity Index Measure (SSIM) values.
  • Efficiency: The Fourier pre-processing step adds only about 1.5 minutes on average to the processing time for a single video, which is a substantial improvement over methods that take over an hour.
  • Robustness: The use of RANSAC makes the method robust to outliers, such as those arising from tracking errors or reflection artifacts.

Limitations and Future Work

  • Conflating Frequencies: While the method handles conflating frequencies by excluding them from the primary correction step, it means that the full deformation might not be recovered in those specific frequency bands.
  • Assumption of Shallow Water: The model assumes a stationary single-plane scene and shallow water devoid of turbidity.
  • Moderate Outliers: The paper notes that the method is robust to "moderate levels of outliers," implying that extreme cases of reflection or blur might still pose challenges.
  • Future Work: The authors suggest that their method is a geometrically and physically-motivated initial step for further distortion removal. This implies that combining it with other advanced techniques is a promising direction. They also note that their method differs from deep learning approaches that only consider single images, suggesting that incorporating temporal information into deep learning models could be beneficial.

Summary updated

BibTeX
@inproceedings{james2020fourier,
  author = {Jerin Geo James and Ajit Rajwade},
  title = {{Fourier Based Pre-Processing For Seeing Through Water}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure