Exploiting Structure Sparsity for Covariance-based Visual Representation

Jianjia Zhang, Lei Wang, Luping ZhouarXiv 2016 · OpenReview

Abstract

The past few years have witnessed increasing research interest on covariance-based feature representation. A variety of methods have been proposed to boost its efficacy, with some recent ones resorting to nonlinear kernel technique. Noting that the essence of this feature representation is to characterise the underlying structure of visual features, this paper argues that an equally, if not more, important approach to boosting its efficacy shall be to improve the quality of this characterisation. Following this idea, we propose to exploit the structure sparsity of visual features in skeletal human action recognition, and compute sparse inverse covariance estimate (SICE) as feature representation. We discuss the advantage of this new representation on dealing with small sample, high dimensionality, and modelling capability. Furthermore, utilising the monotonicity property of SICE, we efficiently generate a hierarchy of SICE matrices to characterise the structure of visual features at different sparsity levels, and two discriminative learning algorithms are then developed to adaptively integrate them to perform recognition. As demonstrated by extensive experiments, the proposed representation leads to significantly improved recognition performance over the state-of-the-art comparable methods. In particular, as a method fully based on linear technique, it is comparable or even better than those employing nonlinear kernel technique. This result well demonstrates the value of exploiting structure sparsity for covariance-based feature representation.

Introduction

Covariance-based feature representation (Cov-RP) is a popular tool in computer vision for capturing the relationships between different features, such as the x, y, and z coordinates of joints in skeletal action recognition. However, Cov-RP faces a significant "small sample, high dimensionality" challenge. In many scenarios, high-dimensional feature vectors are estimated from very few video frames, leading to unstable or singular covariance matrices that degrade recognition performance.

Existing solutions often involve adding a small identity matrix to the covariance or employing nonlinear kernels. However, these methods often ignore crucial prior knowledge: structure sparsity. For example, in a human skeleton, the movement of a hand is directly related to the elbow but only indirectly related to the foot. Standard covariance matrices conflate these direct and indirect relationships. In contrast, the inverse covariance matrix (the precision matrix) isolates direct interactions, providing a cleaner characterization of the data's underlying structure.

Proposed Method: Sparse Inverse Covariance Estimation (SICE)

The authors propose migrating from the standard covariance matrix to its inverse using Sparse Inverse Covariance Estimation (SICE).

SICE-RP

Instead of estimating a standard covariance matrix, the proposed method solves an optimization problem that imposes an $L_1$ penalty on the inverse covariance matrix. This forces "unimportant" correlations to zero, resulting in a sparse representation that highlights critical structural dependencies.

Visualization of the "Crouch or hide" action and its corresponding SICE-RP representation.

Hierarchical Sparsity and Discriminative Learning

Since selecting a single optimal sparsity level ($\lambda$) is difficult, the authors utilize the "monotonicity property" of SICE to efficiently generate a hierarchy of matrices ranging from dense to very sparse. Two algorithms, SICE-RP$\beta$ and SICE-RP$_M$, were developed to adaptively integrate this hierarchy. These algorithms learn to weight different sparsity levels to maximize classification accuracy within a kernel-induced feature space, respecting the Riemannian geometry of the matrices.

Experimental Results

The researchers evaluated their approach on several benchmark skeletal action/gesture datasets, including HDM05, MSRC-12, MSR-Action3D, and MSR-DailyActivity3D, as well as the ADHD-200 medical imaging dataset.

Performance comparison on HDM05 and MSRC-12 datasets showing the superiority of SICE-based methods.

Key findings include:

  • Action Recognition: On the MSR-Action3D dataset, SICE-RP achieved 96.9% accuracy, a significant improvement over the 74.0% achieved by standard Cov-RP.
  • Large Scale Performance: On the 100-class HDM05 experiment, SICE-RP$\beta$ reached 69.3%, outperforming the best nonlinear kernel method (66.2%).
  • Generalization: On the ADHD-200 brain imaging dataset, SICE-RP$_M$ achieved 73.2%, significantly beating the previous state-of-the-art of 69.6%.
  • Efficiency: A $100 \times 100$ SICE matrix can be computed in just 0.014 CPU seconds, demonstrating the method's computational viability.

Conclusion

This research demonstrates that exploiting structural sparsity through SICE-based representations can significantly boost the efficacy of covariance-based features. Notably, this purely linear approach often outperforms complex nonlinear kernel methods, suggesting that better modeling of underlying data structure is a powerful alternative to increasing model complexity. Future work will explore unsupervised learning scenarios and the potential of combining SICE with nonlinear representations.

Summary updated

Abstract

The past few years have witnessed increasing research interest on covariance-based feature representation. A variety of methods have been proposed to boost its efficacy, with some recent ones resorting to nonlinear kernel technique. Noting that the essence of this feature representation is to characterise the underlying structure of visual features, this paper argues that an equally, if not more, important approach to boosting its efficacy shall be to improve the quality of this characterisation. Following this idea, we propose to exploit the structure sparsity of visual features in skeletal human action recognition, and compute sparse inverse covariance estimate (SICE) as feature representation. We discuss the advantage of this new representation on dealing with small sample, high dimensionality, and modelling capability. Furthermore, utilising the monotonicity property of SICE, we efficiently generate a hierarchy of SICE matrices to characterise the structure of visual features at different sparsity levels, and two discriminative learning algorithms are then developed to adaptively integrate them to perform recognition. As demonstrated by extensive experiments, the proposed representation leads to significantly improved recognition performance over the state-of-the-art comparable methods. In particular, as a method fully based on linear technique, it is comparable or even better than those employing nonlinear kernel technique. This result well demonstrates the value of exploiting structure sparsity for covariance-based feature representation.
BibTeX
@misc{zhang2016exploiting,
  author = {Jianjia Zhang and Lei Wang and Luping Zhou},
  title = {{Exploiting Structure Sparsity for Covariance-based Visual Representation}},
  year = {2016},
  eprint = {1610.08619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.08619},
}
Figure