Abstract
This work studies the problem of learning appropriate low dimensional image representations. We propose a generic algorithmic framework, which leverages two classic representation learning paradigms, i.e., sparse representation and the trace quotient criterion. The former is a well-known powerful tool to identify underlying self-explanatory factors of data, while the latter is known for disentangling underlying low dimensional discriminative factors in data. Our developed solutions disentangle sparse representations of images by employing the trace quotient criterion. We construct a unified cost function, coined as the SPARse LOW dimensional representation (SparLow) function, for jointly learning both a sparsifying dictionary and a dimensionality reduction transformation. The SparLow function is widely applicable for developing various algorithms in three classic machine learning scenarios, namely, unsupervised, supervised, and semi-supervised learning. In order to develop efficient joint learning algorithms for maximizing the SparLow function, we deploy a framework of sparse coding with appropriate convex priors to ensure the sparse representations to be locally differentiable. Moreover, we develop an efficient geometric conjugate gradient algorithm to maximize the SparLow function on its underlying Riemannian manifold. Performance of the proposed SparLow algorithmic framework is investigated on several image processing tasks, such as 3D data visualization, face/digit recognition, and object/scene categorization.
Introduction
In the field of computer vision, high-dimensional image data presents a significant challenge for efficient processing and classification. To address this, researchers seek to compress data into low-dimensional representations that retain essential discriminative information while discarding noise. Historically, sparse representation has been a dominant tool for uncovering the underlying structure of images. However, traditional approaches often decouple the learning process: a dictionary is first learned to extract sparse features, which are then fed into a separate dimensionality reduction tool like PCA or LDA.
This decoupled approach is suboptimal as the learned features may not be tailored for the final task. While some "task-driven" dictionary learning methods have emerged, they are often limited to supervised scenarios and scale poorly with the number of classes. This paper introduces SparLow (SPARse LOW dimensional representation), a unified framework that jointly learns a compact dictionary and a task-specific projection simultaneously across unsupervised, supervised, and semi-supervised paradigms.
The SparLow Framework
The SparLow framework is structured as a two-layer interconnected system designed for joint optimization:
- Layer 1: Sparse Coding with Convex Priors. This layer extracts underlying factors of variation. By utilizing convex sparsity priors, such as the Elastic Net, the authors mathematically ensure that the resulting sparse representations are locally differentiable. This differentiability is a critical breakthrough, as it allows for the use of gradient-based methods in a joint optimization setting.
- Layer 2: Trace Quotient (TQ) Criterion. The sparse features are then projected into a lower-dimensional space using the TQ criterion. TQ is a versatile mathematical formulation that generalizes various dimensionality reduction techniques. By adjusting the matrices within the TQ formula, SparLow can adapt to mimic PCA for unsupervised learning, LDA for supervised learning, or graph-based methods for semi-supervised learning.
Optimization on Riemannian Manifolds
To train these layers simultaneously, the framework maximizes a unified cost function. However, standard gradient descent is inapplicable because the projection matrix must remain orthogonal and the dictionary atoms must maintain a unit norm. To overcome this, the authors developed a custom geometric Conjugate Gradient (CG) algorithm.
As illustrated in the figure above, the optimization operates on a Riemannian manifold—specifically, a product of unit spheres and a Grassmann manifold. The algorithm moves from a point $M^{(j)}$ to $M^{(j+1)}$ along the manifold, utilizing parallel transport and tangent spaces to ensure the constraints are satisfied throughout the iterative process.
Experimental Results
The SparLow framework was rigorously evaluated on standard benchmarks, including handwritten digits (USPS, MNIST), faces (CMU-PIE), and complex object/scene categories (Caltech-101, Caltech-256, 15-Scenes).
- Classification Performance: SparLow consistently outperformed state-of-the-art dictionary learning methods like LC-KSVD and FDDL. On the 15-Scenes dataset, supervised SparLow achieved approximately 98.4% accuracy, significantly higher than competing methods.
- Semi-supervised Learning: The framework showed particular strength in scenarios with limited labeled data. By leveraging unlabeled data to build a more robust sparsifying dictionary, semi-supervised SparLow provided massive gains over standard approaches on the Caltech-101 dataset.
- Visual Disentanglement: Visualizations of the CMU-PIE dataset demonstrated that SparLow effectively disentangles factors of variation, such as identity, pose, and illumination, leading to superior clustering in 3D space compared to traditional methods like LLE.
Conclusion
SparLow represents a significant advancement in representation learning by unifying sparse coding and dimensionality reduction into a single, differentiable framework. By balancing data reconstruction with task-specific goals through a "warm start" regularizer and employing sophisticated manifold optimization, the framework achieves high accuracy and robustness across diverse machine learning tasks. Future work will explore extending this framework to broader classes of low-dimensional models beyond the Trace Quotient criterion.
Summary updated