Covariance-free Partial Least Squares: An Incremental Dimensionality Reduction Method

Artur Jordão, Maiko M. I. Lie, Victor Hugo Cunha de Melo, William Robson SchwartzWACV 2021 · OpenReview

Abstract

Dimensionality reduction plays an important role in computer vision problems since it reduces computational cost and is often capable of yielding more discriminative data representation. In this context, Partial Least Squares (PLS) has presented notable results in tasks such as image classification and neural network optimization. However, PLS is infeasible on large datasets, such as ImageNet, because it requires all the data to be in memory in advance, which is often impractical due to hardware limitations. Additionally, this requirement prevents us from employing PLS on streaming applications where the data are being continuously generated. Motivated by this, we propose a novel incremental PLS, named Covariance-free Incremental Partial Least Squares (CIPLS), which learns a low-dimensional representation of the data using a single sample at a time. In contrast to other state-of-the-art approaches, instead of adopting a partially-discriminative or SGD-based model, we extend Nonlinear Iterative Partial Least Squares (NI-PALS) — the standard algorithm used to compute PLS — for incremental processing. Among the advantages of this approach are the preservation of discriminative information across all components, the possibility of employing its score matrices for feature selection, and its computational efficiency. We validate CIPLS on face verification and image classification tasks, where it outperforms several other incremental dimensionality reduction techniques. In the context of feature selection, CIPLS achieves comparable results when compared to state-of-the-art techniques.

Introduction

Dimensionality reduction is a cornerstone of modern machine learning, essential for reducing computational costs and yielding discriminative data representations. While Principal Component Analysis (PCA) is widely used, Partial Least Squares (PLS) is often preferred in classification tasks because it considers the relationship between features and labels, rather than just variance. However, traditional PLS is a "batch" algorithm, requiring all data to be in memory—a significant hurdle for large-scale datasets like ImageNet or real-time streaming applications.

The Challenge of Incremental PLS

Existing incremental dimensionality reduction methods face various limitations. Stochastic Gradient Descent (SGD)-based approaches often suffer from slow convergence and sensitivity to hyperparameter tuning. Previous incremental PLS (IPLS) variants frequently lose discriminative power as more components are extracted, failing to maintain the feature-label relationship across higher dimensions. This drop-off occurs because the relationship between data and labels isn't correctly preserved during the update process.

CIPLS: A Novel Incremental Approach

The authors introduce Covariance-free Incremental Partial Least Squares (CIPLS), which processes data one sample at a time. By extending the Nonlinear Iterative Partial Least Squares (NI-PALS) algorithm—the standard for computing PLS—CIPLS avoids the need for massive covariance matrices. Key technical innovations include:

  • Incremental Centering: Using an approximate incremental mean to center incoming samples.
  • Covariance-free Updates: An algebraic decomposition allows updates to loading and score matrices using only the current sample and previous state.
  • Deflation Process: Subtracting the contribution of existing components to ensure each new dimension captures unique, useful information.
Comparison of data projections using IPLS, SGDPLS, and the proposed CIPLS. CIPLS demonstrates superior separation and alignment of classes compared to existing incremental methods.

Experimental Evaluation

CIPLS was validated on face verification (LFW, YTF) and image classification (ImageNet) tasks. It consistently outperformed other incremental techniques, such as IPLS and SGDPLS. For instance, on the LFW dataset, CIPLS achieved 91.78% accuracy, nearing the 92.47% "gold standard" of batch PLS.

Accuracy performance on the YouTube Faces (YTF) dataset across sequential blocks of samples. CIPLS (black squares) consistently maintains higher accuracy than CCIPCA, SGDPLS, and IPLS.

Notably, CIPLS maintains high accuracy even with many latent components, whereas methods like IPLS degrade significantly after the first few dimensions. In terms of efficiency, CIPLS is orders of magnitude faster than SGD-based methods and comparable in speed to CCIPCA, the industry standard for fast incremental PCA. Furthermore, CIPLS enables real-time feature selection via the Variable Importance in Projection (VIP) metric.

Conclusion

CIPLS provides a robust, efficient solution for applying PLS to streaming and large-scale data. It bridges the gap between batch processing and incremental requirements, offering high discriminative power with minimal computational overhead. By preserving discriminative information across all components and enabling real-time feature selection, CIPLS represents a significant advancement in incremental dimensionality reduction. Future work may explore its integration into deep learning training loops for dynamic feature pruning.

Summary updated

Abstract

Dimensionality reduction plays an important role in computer vision problems since it reduces computational cost and is often capable of yielding more discriminative data representation. In this context, Partial Least Squares (PLS) has presented notable results in tasks such as image classification and neural network optimization. However, PLS is infeasible on large datasets, such as ImageNet, because it requires all the data to be in memory in advance, which is often impractical due to hardware limitations. Additionally, this requirement prevents us from employing PLS on streaming applications where the data are being continuously generated. Motivated by this, we propose a novel incremental PLS, named Covariance-free Incremental Partial Least Squares (CIPLS), which learns a low-dimensional representation of the data using a single sample at a time. In contrast to other state-of-the-art approaches, instead of adopting a partially-discriminative or SGD-based model, we extend Nonlinear Iterative Partial Least Squares (NI-PALS) — the standard algorithm used to compute PLS — for incremental processing. Among the advantages of this approach are the preservation of discriminative information across all components, the possibility of employing its score matrices for feature selection, and its computational efficiency. We validate CIPLS on face verification and image classification tasks, where it outperforms several other incremental dimensionality reduction techniques. In the context of feature selection, CIPLS achieves comparable results when compared to state-of-the-art techniques.
BibTeX
@inproceedings{jordao2021covariance,
  author = {Artur Jordão and Maiko M. I. Lie and Victor Hugo Cunha de Melo and William Robson Schwartz},
  title = {{Covariance-free Partial Least Squares: An Incremental Dimensionality Reduction Method}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=sXZ5fYdHR0},
}
Figure