In traditional machine learning, we assume that a single entity possesses a complete "feature vector" for every individual in a dataset. However, real-world data is often siloed across different organizations. A bank might hold a customer's transaction history, while an e-commerce site holds their shopping preferences. When different entities hold distinct attributes (features) for the same set of individuals, the data is vertically partitioned.
Training a model in this environment is fundamentally different from standard distributed learning. In horizontal partitioning—where different parties have different people but the same features—each party can train a local model and average the results. In vertical partitioning, as Vaidya and Clifton and Clifton established in foundational work, no single participant possesses the complete information necessary to perform even basic global calculations.
The Challenge of Missing Context
The core difficulty of vertical learning is that features are often only meaningful in the context of other features. For example, a model cannot easily calculate the similarity between two individuals if one party only sees their age and another only sees their income. As Huang et al. demonstrate, even standard techniques like Kernel Principal Component Analysis (KPCA) require clever mathematical decomposition—such as exploiting the property that a global RBF kernel matrix is the element-wise product of local ones—to function without pooling raw data.
This structural split creates a "hard dependency." As Qiu points out, the global model's input is a composite; if a single feature provider drops out, the forward pass of the model is effectively broken because the input vector is incomplete.
Learning Through Intermediate Representations
To overcome this, researchers use Vertical Federated Learning (VFL). Instead of sharing raw features, participants compute intermediate representations, such as embeddings or partial gradients, and send them to a central aggregator or a "label party."
For instance, in neural networks, each party might run a "bottom model" on their local features to produce an embedding. These embeddings are then concatenated and fed into a "top model" that holds the labels. He et al. proposed using irreversible linear transformations to protect these layers, allowing for efficient training without the massive overhead of full homomorphic encryption.
However, this exchange introduces a new vulnerability: label leakage. Liu et al. show that the very gradients and embeddings intended to protect privacy can be exploited by an adversarial participant to infer the sensitive labels held by another party. Similarly, Xu et al. note that shared intermediate values can leak information about the underlying features themselves.
Securing the Vertical Pipeline
To make VFL practical, the research community focuses on three pillars: privacy, efficiency, and feature selection.
- Cryptographic Protection: To prevent leakage, researchers combine multiple privacy techniques. Chen et al. (CAESAR) and Tjuawinata et al. (MUSE-VFL) demonstrate that "marrying" Homomorphic Encryption (HE) with Secure Multi-party Computation (MPC) can significantly speed up training by shifting heavy computations to local operations while using MPC only for essential coordination.
- Communication Efficiency: Because VFL requires frequent exchange of high-dimensional embeddings, communication is a bottleneck. Castiglia et al. introduced LESS-VFL, which identifies significant embedding components at the server to prune irrelevant features at the source, reducing communication costs by up to 20x.
- Feature Selection without Visibility: A major hurdle is identifying which features are actually useful when you cannot see the labels or the other features. Li et al. (FedSDG-FS) and Qian et al. address this by developing methods to calculate feature importance and high-order interactions across silos, ensuring that the final model is not just private, but also performant.
By treating the distributed features as a single, virtual dataset, these methods allow organizations to unlock the value of their collective data—such as detecting financial fraud as explored by Arora et al.—without ever compromising the privacy of the individuals they serve.
Key Foundational Work
Privacy-preserving k-means clustering over vertically partitioned data (2003) — contributed 1 concept elements to this direction.
Learning Support Vector Machines from Distributed Data Sources (2005) — contributed 1 concept elements to this direction.