Block Variable Selection in Multivariate Regression and High-dimensional Causal Inference

Aurelie C. Lozano, Vikas SindhwaniNeurIPS 2010 · OpenReview

Introduction

In modern data science, researchers frequently encounter "high-dimensional" problems where the number of potential predictors (inputs) and responses (outputs) far exceeds the number of available observations. Standard regression techniques often struggle in these scenarios, particularly because they treat multiple outputs as independent—ignoring the rich correlations between them that could improve model accuracy. Furthermore, in fields ranging from genomics to social science, variables rarely act in isolation; they belong to functional groups or "blocks."

This paper introduces Multivariate Group Orthogonal Matching Pursuit (MGOMP), a powerful algorithm designed to select groups of relevant variables in high-dimensional regression problems where multiple outputs are correlated. By applying this method to time-series data, the authors develop a novel framework for high-dimensional causal inference. This allows them to identify "key influencers" in networks based on the actual content they produce, rather than relying solely on structural connections like hyperlinks or citations.

Methodology: The MGOMP Algorithm

The core contribution of this work is MGOMP, an extension of the classic Orthogonal Matching Pursuit (OMP) algorithm tailored for complex, grouped data.

  • Block Selection: Unlike standard OMP, which selects one variable at a time, MGOMP performs a "greedy pursuit" of variable blocks. It evaluates groups of input variables against groups of output variables, making it ideal for datasets with inherent categorical or functional groupings.
  • Leveraging Output Correlation: Most variable selection methods ignore the error covariance between outputs. MGOMP incorporates an estimate of the precision matrix (the inverse of the covariance matrix). This allows the algorithm to use the "relatedness" of different outputs to make more robust decisions about which inputs are truly relevant.
  • Granger Causality: The authors frame causal inference as a Vector Autoregressive (VAR) problem. If the past content of one author helps predict the future content of another (above and beyond the second author's own history), a "Granger-causal" link is established. MGOMP efficiently finds these sparse causal links across massive collections of time-series data.
  • GrangerRanks: Based on the resulting causal graph, the authors propose new influence measures, such as GrangerPageRank and GrangerOutDegree, to identify true authorities.

Key Insights and Results

The authors validated their framework using simulated datasets, a dataset of High-Energy Physics papers (KDD Cup 2003), and a real-world crawl of IBM Lotus bloggers.

Output-Centric Learning and Superior Accuracy

A major insight of this research is that the relationship between outputs is a valuable source of information, not just noise. By accounting for how outputs move together, MGOMP can identify relevant predictors that might appear weak when viewed through the lens of a single output.

Simulation results comparing MGOMP variants against Group-OMP and standard OMP.

In simulations with varying levels of noise and output correlation, MGOMP consistently outperformed baseline methods like Group-OMP and standard OMP in both prediction accuracy and variable identification (F1 score). This performance gap widened as the correlation between outputs increased, proving the value of the algorithm's output-centric approach.

Content vs. Connectivity: Validation via Citations

Traditional influence metrics like PageRank rely on the "plumbing" of a network (e.g., hyperlinks or citations). However, a link does not always imply influence—it might be a blogroll link or a citation of a disagreed-upon paper. Causal graphs derived from actual text content are often sparser and more representative of true intellectual influence.

Causal graph of the High-Energy Physics community and a comparison of Granger-based influence rankings versus traditional citation counts.

When applied to the High-Energy Physics dataset, the authors compared their content-derived GrangerPageRank to traditional citation counts. They found a high positive correlation (0.728). The causal model successfully identified the same "authorities" (such as E. Witten and C. Vafa) recognized by the scientific community through formal citations, validating that content-driven causal links can accurately map influence.

Furthermore, in the IBM Lotus study, the causal graph revealed distinct sub-community structures that were entirely invisible or cluttered in the standard hyperlink graph.

Limitations and Future Work

While MGOMP offers a robust solution for high-dimensional causal inference, the authors note a few areas for future development:

  • Dynamic Time-Lags: The current model uses a pre-defined maximum time lag (e.g., 7 days or 12 months). Automatically optimizing this lag for different pairs of variables could yield more precise causal links.
  • Hierarchical Structures: Incorporating hierarchical group structures (groups within groups) could further improve the model's ability to capture complex social dynamics, though this must be balanced against computational complexity.
  • Hybrid Models: Combining the "content-based" causal graph with the "structural" hyperlink graph could create a more holistic view of influence, merging what people say with how they connect.

Summary updated

Abstract

We consider multivariate regression problems involving high-dimensional predictor and response spaces. To efficiently address such problems, we propose a variable selection method, Multivariate Group Orthogonal Matching Pursuit, which extends the standard Orthogonal Matching Pursuit technique to account for arbitrary sparsity patterns induced by domain-specific groupings over both input and output variables, while also taking advantage of the correlation that may exist between the multiple outputs. We illustrate the utility of this framework for inferring causal relationships over a collection of high-dimensional time series variables. When applied to time-evolving social media content, our models yield a new family of causality-based influence measures that may be seen as an alternative to PageRank. Theoretical guarantees, extensive simulations and empirical studies confirm the generality and value of our framework.
BibTeX
@inproceedings{lozano2010block,
  author = {Aurelie C. Lozano and Vikas Sindhwani},
  title = {{Block Variable Selection in Multivariate Regression and High-dimensional Causal Inference}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=SJE3BPbOWH},
}
Figure