Cholesky Decomposition-Based Metric Learning for Video-Based Human Action Recognition

Si Chen, Yuanyuan Shen, Yan Yan, Dahan Wang, Shunzhi ZhuIEEE Access 2020 · DOI

Introduction

Video-based human action recognition is a critical task in computer vision, yet it remains challenging due to environmental "noise" such as fluctuating illumination, camera motion, and complex backgrounds. To address these issues, researchers often employ metric learning, which seeks a mathematical distance metric that minimizes the distance between similar actions while maximizing it for dissimilar ones.

However, traditional metric learning methods face a significant computational hurdle. While full-rank similarity matrices are flexible, they are prone to overfitting. Conversely, low-rank constraints, which mitigate overfitting, typically require Singular Value Decomposition (SVD) at every optimization step. SVD is computationally expensive and iterative, creating a bottleneck for high-dimensional video data. This paper introduces Cholesky Decomposition-based Metric Learning (CDML), a novel approach that enforces low-rank properties through a faster mathematical shortcut, significantly improving training efficiency without sacrificing accuracy.

Methodology

The CDML framework processes video data through a four-stage pipeline designed to extract robust features and learn an optimal similarity metric.

The CDML pipeline for video-based human action recognition.
  1. Feature Detection: The system utilizes Improved Dense Trajectories (iDT) to track points across video frames, capturing essential motion and appearance information.
  2. Preprocessing: White Principal Component Analysis (WPCA) is applied to the extracted features to reduce dimensionality and filter out noise.
  3. Encoding: The Vector of Locally Aggregated Descriptors (VLAD) method aggregates local trajectories into a global, fixed-length vector representation for each video.
  4. The CDML Algorithm: The core innovation lies in the metric learning stage. Instead of learning a similarity matrix $M$ directly, CDML decomposes it using Cholesky Decomposition ($M = UU^T$), where $U$ is a lower triangular matrix. By limiting the number of non-zero columns in $U$, the authors impose a "hard" low-rank constraint. This allows the optimization problem to be solved via standard gradient descent, bypassing the need for expensive SVD operations. Furthermore, a smooth, differentiable generalized logical loss function is used to ensure stable convergence.

Experimental Results

The effectiveness of CDML was evaluated on the ASLAN (Action Similarity Labeling) dataset, which consists of 3,697 YouTube videos across 432 action categories. The task involved action verification—determining if two videos depict the same action type—using a 10-fold cross-validation setup.

CDML achieved a state-of-the-art mean verification rate of 63.3% when combining multiple descriptors, including HOG (appearance), HOF (motion), and MBH (motion boundaries). This performance surpassed several established methods such as LRML, ARL, and MHCCCA. Notably, CDML also demonstrated significant efficiency gains; on single-feature tests using MBH, it reached 60.59% accuracy, far exceeding Euclidean (53.55%) and Cosine (54.89%) baselines. The authors highlighted that CDML requires substantially less training time than other low-rank methods due to the avoidance of iterative SVD.

Discussion and Conclusion

The primary contribution of this work is the transition from a non-convex, NP-hard rank minimization problem to a differentiable gradient descent problem via Cholesky decomposition. This "implicit rank control" is more efficient than the "soft" trace-norm regularization used in previous state-of-the-art methods. The results confirm that combining diverse descriptors provides a robust representation that CDML can effectively optimize.

While CDML shows great promise, it currently relies on hand-crafted features (iDT, VLAD), whereas modern action recognition is increasingly moving toward end-to-end deep learning architectures. Future work could involve integrating Cholesky-based constraints directly into deep learning models and testing the algorithm on larger benchmarks like Kinetics to further validate its scalability in real-world, real-time applications.

Summary updated

BibTeX
@article{chen2020cholesky,
  author = {Si Chen and Yuanyuan Shen and Yan Yan and Dahan Wang and Shunzhi Zhu},
  title = {{Cholesky Decomposition-Based Metric Learning for Video-Based Human Action Recognition}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2966329},
}
Figure