A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks

Yi-Lei Chen, Chiou-Ting HsuICCV 2013 · DOI

Introduction

Rain streaks significantly degrade the performance of computer vision systems, obscuring critical features necessary for tasks such as object detection and tracking. Traditional rain removal techniques often struggle with single-image scenarios or require computationally expensive dictionary learning. This paper presents a unified low-rank appearance model designed to efficiently separate rain streaks from both static images and video sequences by exploiting their inherent mathematical structure.

Methodology: The Low-Rank Appearance Model

The proposed framework leverages the observation that rain streaks exhibit high spatio-temporal correlation. In a given scene, streaks typically fall in similar directions and at comparable speeds, making them highly repetitive across local image patches.

Spatio-temporal correlation of rain streaks across a scene. Colored boxes indicate local patches used for low-rank modeling.

Patch-Based Decomposition

The core of the method involves a "patch map" that rearranges local image patches into a matrix (for 2D images) or a tensor (for 3D video volumes). Because the rain streaks are highly similar, this structure is inherently low-rank. The input image $I$ is modeled as a combination of three components: $I = I_R + I_S + N$, where $I_R$ represents the rain component, $I_S$ is the rain-free background, and $N$ is noise.

Optimization and Enhancement

To solve this decomposition, the authors employ a non-convex rank penalty ($p=0.5$). Unlike the standard nuclear norm ($p=1$), this approach more accurately captures the sparse, low-rank nature of rain without erroneously suppressing background textures. Total Variation (TV) regularization is applied to maintain background sharpness. Finally, a Guided Image Filter (GIF) is used to restore fine details that may be lost during the low-rank approximation, ensuring the final output remains sharp.

Experimental Evaluation

The model was evaluated using synthetic images (Kang et al. dataset) and real-world video sequences (e.g., "heavy rain," "mag," and "forrest"). Performance was quantitatively measured using the Visual Information Fidelity (VIF) metric to assess the preservation of natural image information.

Results and Efficiency

The proposed method consistently outperformed state-of-the-art dictionary-learning techniques in both accuracy and speed. While previous methods required 8–22 seconds per image (excluding training time), the low-rank model processed images in approximately 3.2 seconds using MATLAB. Qualitatively, the method preserves global structures and complex background features—such as facial details—better than simple filtering or detection-based approaches.

Discussion and Limitations

A primary challenge for the low-rank model is the "low-rank trap": repetitive background textures, such as brick walls or fences, can be mistaken for rain streaks due to their regular patterns. The authors suggest that incorporating orientation-based features, such as Histogram of Oriented Gradients (HOG), could help the model distinguish between static architectural patterns and the specific orientation of falling rain.

Conclusion

By treating rain removal as a signal separation problem rather than a pixel-wise detection task, this low-rank appearance model provides a fast, versatile, and effective solution for enhancing visual data in inclement weather. Its ability to generalize across images and video volumes makes it a robust tool for real-world computer vision applications.

Summary updated

BibTeX
@inproceedings{chen2013generalized,
  author = {Yi-Lei Chen and Chiou-Ting Hsu},
  title = {{A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  doi = {10.1109/iccv.2013.247},
}
Figure