Symphony in the Latent Space: Provably Integrating High-Dimensional Techniques with Non-linear Machine Learning Models

Qiong Wu, Jian Li, Zhenming Liu, Yanhua Li, Mihai Cucuringu

TL;DR

This paper introduces the Additive Influence Model, a framework that successfully "decouples" the challenge of mapping high-dimensional entity interactions (like how one stock affects another) from the challenge of modeling non-linear features. By separating these tasks, the authors provide a method that enjoys both the theoretical rigor of high-dimensional statistics and the predictive "oomph" of modern non-linear machine learning, significantly outperforming standard models in financial and social network forecasting.

What Problem Does This Paper Address?

Predicting the future state of a system with many moving parts—like the stock market or a social network—is notoriously difficult because of two conflicting requirements:

  1. Cross-entity interactions (High Dimensionality): In a system with 3,000 entities, there are nearly 10 million potential pairwise interactions. However, we usually have far fewer data points (e.g., only a few thousand days of historical data). This leads to massive overfitting if you try to model every possible connection.
  2. Feature-response interactions (Non-linearity): The relationship between a feature (like a company's revenue) and the outcome (its stock price) is rarely a straight line. We need complex models like Neural Networks or Gradient Boosted Trees to capture these patterns.

Existing "ad-hoc" solutions usually try to shove both problems into a single black box (e.g., a Neural Network with a Lasso penalty). The authors argue that this approach is fragile: the mathematical guarantees of the statistical tools often break when combined with non-linear models, leading to poor performance in real-world, time-evolving systems.

Methods

The core innovation is the Additive Influence Model, which assumes the system follows a specific generative structure: $$y_{t,i} = \sum_{j \le d} \kappa(z_i, z_j) g(x_{t,j}) + \xi_{t,i}$$ Where $y$ is the prediction, $\kappa$ is a kernel function measuring the "distance" between entities in a latent space, and $g(\cdot)$ is a non-linear function of the features.

The authors prove that you can decouple the learning process into two distinct stages:

  • Stage 1: Learning the Latent Space. They use a kernel-based technique to embed entities into a low-dimensional space. By using a low-rank approximation of the covariance matrix, they can provably estimate the interaction matrix $K$ without needing to know the non-linear function $g(\cdot)$ beforehand.
  • Stage 2: Learning the Non-linearity. Once the interaction structure is fixed, they apply specialized ML techniques to learn $g(\cdot)$. They introduce two new algorithms:
  • nparam-gEST: A moment-based, non-parametric method with statistical guarantees.
  • LIN-PVEL (Linear Projected Vector Ensemble Learner): A gradient boosting approach that uses "linearized" versions of decision trees to maintain computational efficiency while capturing complex feature interactions.

Datasets

The researchers evaluated their framework on two primary domains:

  1. Equity Market Data: 10 years of daily data from an emerging market. They tested on two "universes": a subset of 800 liquid stocks (similar to the S&P 500) and a "Full Universe" of all stocks. They used 337 technical factors as features.
  2. Social Network Data: Twitter data used to predict the future popularity of accounts based on user interactions (discussed in the appendix, with summary results in the main text).

Key Insights and Ideas

  • Geometry as a Regularizer: By forcing entity interactions to exist in a Euclidean latent space, the model's degrees of freedom drop from $O(d^2)$ to something much smaller. If entity A is close to B, and B is close to C, then A must be close to C. This geometric constraint prevents the model from "hallucinating" random, spurious connections.
  • The Power of Decoupling: The "aha moment" is that you don't need to solve the whole problem at once. You can find the "who affects whom" structure (the latent space) using purely statistical methods, then hand that structure over to a "black box" ML model to figure out the "how they affect each other" (the non-linearity).
  • Linearized Trees: For high-dimensional vector regressions, standard decision trees are too slow. The authors show that "unfolding" shallow trees into a sum of interaction terms (linearization) provides a massive speedup without sacrificing the ability to model smooth non-linear changes.

Key Results

The proposed models, particularly LIN-PVEL, dominated the baselines across almost all metrics:

  • Forecasting Power: In the "Full Universe" equity test, the consolidated model achieved a correlation of 0.0958, significantly higher than standard MLPs (0.0606) or LSTMs (0.0352).
  • Investment Performance: In backtesting (Profit & Loss), the LIN-PVEL model achieved a Sharpe Ratio of 10.97, nearly double that of a standard MLP (6.38) and far exceeding existing cross-entity models like ARRR (3.24).
  • Robustness: The models maintained high t-statistics (significance), suggesting the results weren't just due to luck or specific market regimes.

Limitations and Future Work

  • Assumptions on Data Distribution: The theoretical proofs rely on "near-uniform" distributions and i.i.d. (independent and identically distributed) samples, which may not perfectly hold in highly volatile or trending markets.
  • Smoothness vs. Discontinuity: The linearized approach in LIN-PVEL is optimized for "smooth" changes. If the underlying relationship has sharp, jagged discontinuities, a traditional deep decision tree might still have an edge, albeit at a much higher computational cost.
  • Dynamic Latent Spaces: The current model assumes the latent positions ($z_i$) are relatively stable. Future work could explore "evolving" latent spaces where the relationships between entities shift over time.

Summary updated

Abstract

This paper revisits building machine learning algorithms that involve interactions between entities, such as those between financial assets in an actively managed portfolio, or interactions between users in a social network. Our goal is to forecast the future evolution of ensembles of multivariate time series in such applications (e.g., the future return of a financial asset or the future popularity of a Twitter account). Designing ML algorithms for such systems requires addressing the challenges of high-dimensional interactions and non-linearity. Existing approaches usually adopt an ad-hoc approach to integrating high-dimensional techniques into non-linear models and recent studies have shown these approaches have questionable efficacy in time-evolving interacting systems. To this end, we propose a novel framework, which we dub as the additive influence model. Under our modeling assumption, we show that it is possible to decouple the learning of high-dimensional interactions from the learning of non-linear feature interactions. To learn the high-dimensional interactions, we leverage kernel-based techniques, with provable guarantees, to embed the entities in a low-dimensional latent space. To learn the non-linear feature-response interactions, we generalize prominent machine learning techniques, including designing a new statistically sound non-parametric method and an ensemble learning algorithm optimized for vector regressions. Extensive experiments on two common applications demonstrate that our new algorithms deliver significantly stronger forecasting power compared to standard and recently proposed methods.
BibTeX
@inproceedings{wu2023symphony,
  author = {Qiong Wu and Jian Li and Zhenming Liu and Yanhua Li and Mihai Cucuringu},
  title = {{Symphony in the Latent Space: Provably Integrating High-Dimensional Techniques with Non-linear Machine Learning Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26233},
}

Loading related content...

Figure