Abstract
Sparsity in the eigenvectors of signal covariance matrices is exploited in this paper for compression and denoising. Dimensionality reduction (DR) and quantization modules present in many practical compression schemes, such as transform codecs, are designed to capitalize on this form of sparsity and achieve improved reconstruction performance compared to existing sparsity-agnostic codecs. Using training data that may be noisy, a novel sparsity-aware linear DR scheme is developed to fully exploit sparsity in the covariance eigenvectors and form noise-resilient estimates of the principal covariance eigenbasis. Sparsity is effected via norm-one regularization, and the associated minimization problems are solved using computationally efficient coordinate descent iterations. The resulting eigenspace estimator is shown capable of identifying a subset of the unknown support of the eigenspace basis vectors even when the observation noise covariance matrix is unknown, as long as the noise power is sufficiently low. It is proved that the sparsity-aware estimator is asymptotically normal, and the probability to correctly identify the signal subspace basis support approaches one as the number of training data grows large. Simulations using synthetic data and images corroborate that the proposed algorithms achieve improved reconstruction quality relative to alternatives.
Introduction
In modern signal processing, Dimensionality Reduction (DR) serves as the foundational "workhorse" for representing complex, high-dimensional signals with minimal bitrates. Standard Principal Component Analysis (PCA) is the conventional choice for this task, projecting data onto a subspace spanned by the principal eigenvectors of the signal's covariance matrix. However, standard PCA suffers from two critical limitations: it is "sparsity blind," failing to exploit the fact that many real-world signals (e.g., in genomics or specialized imaging) possess sparse eigenvectors, and it is highly sensitive to noise. When training data is corrupted by "colored" (non-white) noise with unknown statistics, standard PCA often fails to identify the underlying signal subspace.
This paper introduces a novel framework that bridges these gaps by developing a DR scheme that is both sparsity-aware and noise-resilient. By integrating $\ell_1$-regularization into the PCA framework, the authors propose "S-PCA" and a corresponding Sparsity-Aware Transform Coder (SATC) designed specifically for optimal reconstruction.
Methodology
Sparse PCA (S-PCA) for Reconstruction
Unlike previous iterations of sparse PCA designed for data interpretation or variable selection, this approach prioritizes minimizing reconstruction error. The core of the method lies in a regularized cost function that augments the standard PCA objective with two penalties:
- $\ell_1$-norm penalty: Induces sparsity in the eigenvectors.
- $\ell_2$-norm penalty: Ensures that the compression and reconstruction matrices remain closely aligned.
Optimization Algorithms
To solve the resulting minimization problems, the authors developed two computationally efficient approaches:
- BCD-SPCA: A Block Coordinate Descent method that iteratively optimizes the compression and reconstruction matrices.
- ECD-SPCA: An Element-wise Coordinate Descent algorithm that provides closed-form updates for each entry. This significantly reduces computational overhead, making the framework more practical for large datasets.
Sparsity-Aware Transform Coder (SATC)
The S-PCA framework is integrated with a Vector Quantizer (VQ) to form the SATC, a complete digital compression pipeline. To ensure robustness, a 5-fold cross-validation scheme is employed to automatically tune the sparsity-controlling hyperparameters ($\lambda$).
Theoretical Insights
The "Oracle" Property and Noise Resilience
A significant contribution of this work is the mathematical proof of the "Oracle" property: as the volume of training data increases, the estimator identifies the true support (the indices of non-zero elements) of the signal subspace with a probability approaching one. Furthermore, the S-PCA estimator demonstrates remarkable resilience to noise. It can accurately identify the signal subspace even when the observation noise covariance is unknown, provided the noise power remains sufficiently low. This offers a distinct advantage over traditional joint denoising-reconstruction schemes that require prior knowledge of noise statistics.
Asymptotic Normality
The paper establishes a solid statistical foundation for the sparsity-aware estimator by proving its asymptotic normality. This theoretical grounding ensures that the estimator behaves predictably as the sample size grows, reinforcing its reliability for high-stakes signal processing tasks.
Experimental Results
The researchers evaluated the proposed algorithms using both synthetic data and high-resolution NASA Martian Terrain images. The Martian dataset is particularly relevant as its localized features result in sparse representations in the transform domain.
- Denoising Performance: In scenarios with a 15dB Signal-to-Noise Ratio (SNR), the SATC achieved a reconstructed image SNR ($SNR_{im}$) of 15.89 dB. This outperformed standard PCA (14.2 dB) and traditional DCT-based reconstruction (13.6 dB).
- Visual Fidelity: Qualitative assessments indicate that SATC produces reconstructed images that are visually superior to traditional codecs at identical bit rates, retaining sharper details and fewer artifacts.
Conclusion and Future Work
This research demonstrates that eigenvector sparsity is a powerful prior for improving the Mean Square Error (MSE) of reconstructed signals. While the current ECD-SPCA algorithm is efficient, future work may focus on further optimizing it for real-time, high-dimensional video applications. Additionally, while the current model assumes linear subspaces and stationary signal statistics, the authors are exploring extensions into non-linear manifolds and nonstationary processes with temporal correlation.
Summary updated