Abstract
This paper revisits building machine learning algorithms that involve interactions between entities, such as those between financial assets in an actively managed portfolio, or interactions between users in a social network. Our goal is to forecast the future evolution of ensembles of multivariate time series in such applications. Designing ML algorithms for such systems requires addressing the challenges of high-dimensional interactions and non-linearity. Existing approaches usually adopt an ad-hoc approach to integrating high-dimensional techniques into non-linear models, which often lack theoretical guarantees and efficacy. We propose the additive influence model, which allows for decoupling the learning of high-dimensional interactions from non-linear feature interactions. We leverage kernel-based techniques with provable guarantees to embed entities in a low-dimensional latent space and generalize machine learning techniques for vector regressions. Extensive experiments demonstrate that our algorithms deliver significantly stronger forecasting power compared to standard and recently proposed methods.
Introduction
Predicting the evolution of complex systems, such as stock markets or social networks, necessitates solving two conflicting problems simultaneously. First, one must account for high-dimensional cross-entity interactions. For instance, a portfolio of 3,000 stocks involves nearly 10 million potential pairwise interactions, often with limited historical data, creating an overparameterized environment prone to overfitting. Second, the feature-response interactions in these systems are typically non-linear.
Existing solutions often rely on ad-hoc integrations, such as applying $L_1$ regularization to neural networks. However, these methods frequently lack theoretical guarantees. The mathematical properties of provable high-dimensional techniques often break when combined with black-box non-linear models. This paper introduces a framework to bridge this gap by decoupling these two learning tasks.
The Additive Influence Model
The core of our proposed framework is the Additive Influence Model. It represents the behavior of an entity $i$ at time $t+1$ as a sum of influences from other entities $j$, passed through a non-linear function $g(\cdot)$: $$y_{t+1,i} = \sum_{j \leq d} \kappa(z_i, z_j)g(x_{t,j}) + \xi_{t,i}$$ Here, $\kappa$ is a kernel measuring similarity in a latent space $z$, and $g(\cdot)$ represents the non-linear signal.
Our approach utilizes a three-step decoupled pipeline:
- Provable Latent Embedding: We estimate the interaction kernel matrix $K$ without prior knowledge of the non-linear function $g(\cdot)$.
- Non-Parametric Learning (nparam-gEST): A moment-based algorithm estimates $g(\cdot)$ with provable statistical bounds, transforming the complex interaction problem into a manageable regression.
- Vector Boosting (LIN-PVEL): A specialized gradient boosting algorithm optimized for vector regressions, using linearized interaction terms to model continuous changes more effectively than discrete decision trees.
Theoretical Foundations and Key Insights
A major technical hurdle in high-dimensional learning is the "small eigenvalue blow-up" problem, where standard matrix square roots become unstable as singular values approach zero. We address this by targeting a specific spectral gap and utilizing a rank-$r$ approximation of the covariance matrix.
By decoupling the learning of entity relationships from feature signals, we prevent the non-linear learner from being overwhelmed by the curse of dimensionality. Furthermore, we found that continuous interaction terms often outperform the discrete splits of traditional decision trees in interacting systems, suggesting that model "smoothness" is a critical feature for these applications.
Experimental Evaluation
We evaluated the framework on two large-scale real-world scenarios:
- Chinese Equity Market: Using 10 years of daily data for ~3,600 stocks and 337 technical factors. Our consolidated model achieved a Sharpe ratio of 11.30, significantly outperforming standard VAR models (3.24) and deep learning baselines like AlphaStock (0.10).
- Twitter Social Network: Predicting the popularity of 2,000 active users based on 804 million tweets. Our method achieved the lowest out-of-sample Mean Squared Error (MSE) and the highest correlation, outperforming Node2Vec and XGBoost.
The results indicate that the "provable" nature of our embedding step successfully mitigates overfitting, as evidenced by the smaller gap between training and testing performance compared to baselines.
Conclusion
The Additive Influence Model provides a theoretically sound bridge between high-dimensional interaction modeling and non-linear machine learning. While current theoretical analysis assumes near-uniform distributions, the empirical success on real-world financial and social data is compelling. Future work will explore extending these proofs to heavy-tailed distributions and dynamically shifting latent spaces.
Summary updated