Abstract
Video representation is an important and challenging task in the computer vision community. In this paper, we assume that image frames of a moving scene can be modeled as a Linear Dynamical System. We propose a sparse coding framework, named adaptive video dictionary learning (AVDL), to model a video adaptively. The developed framework is able to capture the dynamics of a moving scene by exploring both sparse properties and the temporal correlations of consecutive video frames. The proposed method is compared with state of the art video processing methods on several benchmark data sequences, which exhibit appearance changes and heavy occlusions.
Introduction
Modeling "Dynamic Textures" (DTs)—stochastic processes like flickering flames, flowing water, or swaying trees—is a complex task in computer vision. Traditional methods often struggle because these scenes lack smooth, parametric motion. The current standard, the Linear Dynamical System (LDS), models video frames as observations of a hidden state evolving over time. However, LDS typically relies on Principal Component Analysis (PCA) to define these states, making it highly sensitive to non-Gaussian noise, such as real-world occlusions or missing data.
Methodology
The Adaptive Video Dictionary Learning (AVDL) framework addresses these limitations by replacing global PCA-based states with sparse coefficients learned over an adaptive dictionary. In this model, each frame $y$ is represented as a sparse combination of atoms from a learned dictionary ($y = Dx$), where the sparse coefficients ($x$) serve as the hidden states.
The framework employs a joint optimization cost function that simultaneously minimizes:
- Transition Error: How well the next state can be predicted from the current one ($x_{i+1} \approx Ax_i$).
- Reconstruction Error: The accuracy of the dictionary and states in reconstructing original frames.
- Sparsity: An "Elastic Net" penalty (a mix of $\ell_1$ and $\ell_2$ norms) to ensure stable, sparse states.
To ensure optimality, the dictionary $D$ is constrained to have unit-norm columns and is optimized as a point on a smooth manifold using Riemannian gradient descent.
Experimental Results
AVDL was evaluated using the DynTex++ benchmark and real-world video sequences. A primary advantage of the sparse representation is its ability to act as a natural filter; since occlusions do not match the repetitive texture learned by the dictionary, they are effectively ignored during state estimation.
In experiments with a burning candle video where 50 frames were corrupted by a synthetic occlusion, AVDL's synthesized output was only 9.47% affected by the artifact, whereas the standard LDS model was 87.01% corrupted. Quantitatively, AVDL significantly outperforms the LDS-NN baseline in recognition tasks, particularly as the occlusion rate increases.
As shown in the table above, at a 30% occlusion rate, AVDL maintained a recognition rate of 22.36%, compared to just 14.17% for the LDS-NN baseline. AVDL also achieved lower reconstruction and transition errors while maintaining a lower compression rate.
Conclusion
AVDL provides a robust framework for modeling dynamic textures by combining the temporal modeling of LDS with the noise-filtering properties of sparse representation. While the method introduces higher computational complexity due to manifold optimization and sparse coding iterations, its resilience to heavy occlusion and non-Gaussian noise represents a significant advancement over traditional PCA-based methods. Future research will focus on improving scalability for high-resolution video sequences.
Summary updated