Localized Sliced Inverse Regression

Qiang Wu, Sayan Mukherjee, Feng LiangNeurIPS 2008 · OpenReview

TL;DR

Localized Sliced Inverse Regression (LSIR) improves upon traditional supervised dimension reduction by incorporating local neighborhood information. By replacing global class means with local averages, LSIR avoids the "degeneracy" problem where standard methods fail to find more than one useful direction in binary classification, and it excels at handling complex, multimodal data.

The Problem with Standard Dimension Reduction

In high-dimensional data analysis, researchers often seek a low-dimensional "subspace" that captures the relationship between explanatory variables ($X$) and a response variable ($Y$). A classic tool for this is Sliced Inverse Regression (SIR), which works by "slicing" the data based on the values of $Y$ and analyzing the means of those slices.

However, SIR suffers from two major limitations:

  1. Degeneracy: In binary classification (where $Y$ has only two values), SIR can only ever find one meaningful direction. If the data has a complex structure requiring multiple directions to explain, SIR fails.
  2. Global Blindness: SIR uses the global mean of each slice. If a single class is "multimodal" (e.g., consisting of two separate clusters in space), the global mean falls in the empty space between them, failing to describe the actual data distribution.

The authors aim to bridge the gap between Manifold Learning (which captures local geometry well but is usually unsupervised) and Supervised Dimension Reduction (which uses labels but often ignores local geometry).

The LSIR Approach

The core innovation of this paper is the localization of the SIR algorithm.

Instead of calculating the mean of all points in a slice like standard SIR, LSIR calculates a local mean ($m_{i,loc}$) for every single data point $x_i$ based only on its $k$-nearest neighbors within the same slice.

The algorithm follows these steps:

  1. Estimate total covariance: Calculate the overall variance of the data.
  2. Find local slice means: For each point, identify its $k$ neighbors in the same class and average them.
  3. Compute localized between-slice covariance: This matrix ($\hat{\Gamma}_{loc}$) uses the local means instead of global ones. Because every point now has its own "local mean," the rank of this matrix is no longer limited by the number of classes, effectively solving the degeneracy problem.
  4. Solve the Eigenproblem: Find the directions (eigenvectors) that maximize the ratio of this local variance to the total variance.
Comparison of embeddings showing LSIR and semiLSIR successfully separating multimodal classes compared to standard principal components.

The authors also introduce a semi-supervised version (semiLSIR). If unlabeled data is available, it is included in the $k$-nearest neighbor search for every slice. This allows the algorithm to learn the underlying shape of the data even where labels are missing.

Key Insights and Applications

The paper validates LSIR using a mix of synthetic "torture tests" and real-world benchmarks, including the Swiss Roll, a complex Tai Chi figure, MNIST digits, and Leukemia gene expression data.

  • Rank Liberation: By using local means, the "between-group" covariance matrix is no longer restricted to a rank of $C-1$ (where $C$ is the number of classes). This allows LSIR to find multiple predictive directions even in simple "A vs B" classification tasks.
  • Automatic Subclass Discovery: Because LSIR looks at local neighborhoods, it naturally identifies if a single class is actually made of multiple sub-clusters. For example, in the Leukemia dataset, LSIR automatically clustered ALL samples into T-cell and B-cell subtypes without prior labels—a structure standard SIR completely missed.
  • Robustness to Noise: Experiments showed that while pure manifold learning might get "lost" in noise, LSIR stays focused on the dimensions that actually correlate with the response variable $Y$.

Key Results

LSIR demonstrates significant performance improvements over existing methods across various tasks:

  • Tai Chi Accuracy: On a complex 2D shape embedded in 6D space, LSIR achieved 98.6% accuracy in retrieving the correct subspace, outperforming Local Fisher Discriminant Analysis (LFDA) at 82%.
  • MNIST Performance: LSIR consistently lowered classification error rates compared to SIR across all digits, with dramatic improvements for "difficult" digits like 2, 3, and 5 that feature complex local structures.
  • Semi-Supervised Gains: In simulations with only 20 labeled points out of 400, semi-supervised LSIR maintained an accuracy of ~0.95, while supervised methods like SAVE and pHd hovered around 0.34 (barely better than random).
Accuracy comparison of LSIR against SIR, SAVE, and PHD across varying sample sizes.

Limitations and Future Work

While powerful, LSIR has a few limitations:

  • The $k$ Parameter: Like many neighborhood-based methods, LSIR depends on the choice of $k$ (the number of neighbors). While values between 10 and 20 usually work well, a more rigorous, automated way to choose $k$ is needed.
  • Computational Complexity: Calculating $k$-NN for every point can be computationally intensive for massive datasets compared to simpler competitors.
  • Kernelization: The authors note that LSIR can be extended using the "kernel trick" to find non-linear directions, leaving the full exploration of "Kernel LSIR" for future research.

Summary updated

Abstract

We developed localized sliced inverse regression for supervised dimension reduction. It has the advantages of preventing degeneracy, increasing estimation accuracy, and automatic subclass discovery in classification problems. A semisupervised version is proposed for the use of unlabeled data. The utility is illustrated on simulated as well as real data sets.
BibTeX
@inproceedings{wu2008localized,
  author = {Qiang Wu and Sayan Mukherjee and Feng Liang},
  title = {{Localized Sliced Inverse Regression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=By-EFPZOWS},
}
Figure