A PLS-Integrated Lasso Method With Application in Index Tracking

Shiqin Tang, Yining Dong, S. Joe Qin

TL;DR

This paper introduces "PLS-integrated Lasso" (PLS-Lasso), a novel regression technique that unifies dimension reduction and variable selection. By integrating Partial Least Squares (PLS) with the Lasso penalty, PLS-Lasso aims to improve predictive performance and sparsity in high-dimensional datasets, demonstrating promising results in financial index tracking.

What Problem Does This Paper Address?

In multivariate data analysis, dimension reduction and regression are often treated as separate steps. Traditional methods like Principal Component Regression (PCR) and Partial Least Squares (PLS) first reduce dimensions and then perform regression. While effective, this separation can lead to suboptimal solutions. The Lasso, a popular regularization technique, excels at variable selection but doesn't inherently incorporate dimension reduction. This paper addresses the challenge of simultaneously performing dimension reduction and variable selection in a unified framework to achieve better predictive accuracy and sparsity, particularly in scenarios with many correlated features.

Methods

The core innovation is the "PLS-integrated Lasso" (PLS-Lasso) method, which merges the principles of PLS and Lasso. The authors propose two formulations:

  • PLS-Lasso-v1: This formulation blends the objectives of PLS (maximizing covariance with the response) and Lasso (minimizing prediction error with L1 regularization) using a multi-objective optimization approach via scalarization. It introduces a new hyperparameter ($\mu$) to balance these objectives. The optimization problem is then solved using iterative algorithms like the Iterative Shrinkage-Thresholding Algorithm (ISTA) or the Douglas-Rachford method.
  • PLS-Lasso-v2: This formulation aims to eliminate the extra hyperparameter $\mu$ from PLS-Lasso-v1. It reformulates the problem using a Charnes-Cooper transformation to handle the quasi-convex objective function. An iterative scheme, often employing the Alternating Direction Method of Multipliers (ADMM), is used to solve the transformed optimization problem.

Both formulations aim to find a sparse weight vector that not only minimizes prediction error but also implicitly performs dimension reduction by focusing on directions that maximize covariance with the target variable.

Datasets

The authors evaluate their proposed methods on two real-world financial datasets:

  • NASDAQ 100 index dataset: Includes weekly stock returns for 82 companies and the NASDAQ 100 index over 596 weeks.
  • S&P 500 index dataset: Includes weekly stock returns for 442 companies and the S&P 500 index over 595 weeks.

The data is split into training (first 400 data points) and testing sets. The primary goal is to predict market index returns using a sparse linear combination of asset returns.

Key Insights and Ideas

  • Unified Framework: The most significant insight is the successful integration of dimension reduction (via PLS principles) directly into the variable selection process (via Lasso). This contrasts with traditional two-step approaches.
  • Pareto Optimality: PLS-Lasso-v2, in particular, is shown to consistently achieve Pareto optimality, meaning it offers better or equal performance across multiple metrics (e.g., lower testing error, higher sparsity) compared to standard Lasso.
  • Hyperparameter Sensitivity: PLS-Lasso-v1's performance can be sensitive to its hyperparameters, suggesting that careful tuning is required for optimal results, whereas PLS-Lasso-v2 offers a more stable solution by reducing the hyperparameter burden.
  • Improved Sparsity-Accuracy Trade-off: The integrated approach allows for achieving higher sparsity levels while maintaining or improving predictive accuracy, which is crucial in high-dimensional financial modeling.

Key Results

The paper demonstrates the efficacy of PLS-Lasso through experiments on financial index tracking.

  • Figure 1 (comparing Lasso, PLS-Lasso-v1 with different $\mu$, and PLS-Lasso-v2) shows that PLS-Lasso-v2 consistently outperforms Lasso by achieving lower training and testing errors at higher sparsity levels.
  • PLS-Lasso-v1 shows competitive performance, especially at higher sparsity levels, but its performance can be less stable than PLS-Lasso-v2 as sparsity decreases.
  • The authors highlight that PLS-Lasso-v2 "consistently demonstrates Pareto optimality over Lasso in terms of lower testing and training errors and higher sparsity."

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Univariate Response Variables: Current PLS-Lasso formulations are designed for univariate response variables. Extending them to handle multivariate responses would broaden their applicability.
  • Single Latent Direction: The current methods focus on extracting only the first latent direction. Future work could explore methods to extract multiple latent components, similar to traditional PLS.
  • Probabilistic Interpretation: PLS-Lasso currently lacks a probabilistic interpretation, which could be a valuable area for future research to deepen theoretical understanding.
  • Generalized Penalty Functions: The L1 penalty could be replaced with other generalized penalty functions (e.g., Lp norms with $0 < p < 1$, or concave penalties) to explore different sparsity patterns and potentially improve performance.

Summary updated

BibTeX
@inproceedings{tang2024pls,
  author = {Shiqin Tang and Yining Dong and S. Joe Qin},
  title = {{A PLS-Integrated Lasso Method With Application in Index Tracking}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10448273},
}
Figure