In traditional machine learning, we assume all data attributes for a given sample are collected in a single centralized matrix. When privacy constraints or data silos prevent centralizing this information, standard Horizontal Federated Learning (HFL) allows multiple institutions to train a model collaboratively, provided they share the exact same feature space but hold data on different individuals.
However, real-world data alliances frequently encounter a completely different structural challenge: multiple organizations share a common user base but hold entirely disjoint sets of attributes or modalities for those users. As formalized by Yang et al., this paradigm is known as vertical federated learning (VFL) or feature-partitioned federated learning.
Consider cross-sector collaborations where data is fragmented across different domains, such as banking and retail entities tracking the same customer base (Li et al.). In healthcare, one clinic might hold clinical imaging while another holds genomic sequences for the same patient population (Chowdhury et al.; Peng et al.). Similarly, in geoenergy, predicting well productivity requires synthesizing geological data held by one firm with operational parameters held by another (Peng et al.). In all these scenarios, foundation models cannot inherently bridge these data partitions without collaborative training frameworks (Baracaldo).
The Mathematical Bottleneck of Disjoint Features
To understand why VFL requires a fundamentally different architecture than standard federated learning, we must look at how data is partitioned. As noted by Liu et al., standard distributed learning strategies assume data is split by instances. In that horizontal regime, every node can independently compute a local loss and local gradients because each node possesses a complete set of features and the corresponding ground-truth labels.
In a vertically partitioned regime, data is split by variables. Suppose Party A holds feature vector $x_1$, Party B holds feature vector $x_2$, and only Party B holds the label $y$. In a standard neural network layer, the activation $z$ depends on the complete feature vector:
$$z = \sigma(W_1 x_1 + W_2 x_2 + b)$$
Because the features $x_1$ and $x_2$ are siloed on separate servers, neither party can compute the forward pass independently. Furthermore, because Party A lacks the label $y$, it cannot compute a loss function or initiate backpropagation. Standard model-averaging algorithms (like Federated Averaging) completely break down because there is no uniform global model architecture that can be evaluated locally on a single party's data.
Step 1: Aligning Entities Securely
Before any mathematical computation can occur, the distributed data silos must determine which rows in their respective databases belong to the same entities. This process is called sample alignment (Yang et al.; Rauniyar et al.).
Party A (Features X1) Party B (Features X2 + Labels Y)
[ ID: 001, Age: 34 ] [ ID: 001, Credit: High, Y: 1 ]
[ ID: 002, Age: 45 ] [ ID: 003, Credit: Low, Y: 0 ]
\ /
\ /
[ Secure Alignment (PSI) ]
|
Matched Sample: ID 001
To prevent organizations from exposing their entire user base or revealing non-overlapping entities, VFL frameworks deploy secure alignment protocols, such as Private Set Intersection (PSI) (Tong et al.; Du). These cryptographic protocols allow parties to identify common sample identifiers (like hashed email addresses or patient IDs) without disclosing the raw identifiers of unmatched records or any underlying feature values. Once aligned, the parties reorder their local datasets to ensure that index $i$ on Party A's server corresponds precisely to index $i$ on Party B's server.
Step 2: The Split Forward and Backward Pass
Once the datasets are aligned, VFL bypasses the feature-silo bottleneck by splitting the machine learning model vertically into local "bottom models" and a shared "top model" (Li et al.).
The Forward Pass: Exchanging Embeddings
Instead of sharing raw features, each participant passes its local attributes through its own bottom model to generate a lower-dimensional representation, or embedding.
- Local Computation: Party A computes an embedding $\mathbf{h}_A = f_A(x_1; \theta_A)$. Concurrently, Party B computes $\mathbf{h}_B = f_B(x_2; \theta_B)$.
- Transmission: These intermediate embeddings—which compress the raw features into abstract representations—are transmitted to the party holding the labels (or a trusted server).
- Aggregation: The label holder combines the embeddings (e.g., via concatenation or summation) and passes them through a top model $g(\mathbf{h}_A, \mathbf{h}_B; \theta_{top})$ to yield the final prediction $\hat{y}$.
Because only abstract embeddings are transmitted, the raw feature values $x_1$ and $x_2$ never leave their local jurisdictions, preserving data sovereignty (Li et al.).
Party A (Features X1) Party B (Features X2 + Labels Y)
| |
v v
[Bottom Model f_A] [Bottom Model f_B]
| |
v (Embedding h_A) v (Embedding h_B)
+------------------> [ Top Model g ] <-------+
|
v
Compute Loss(y, ŷ)
The Backward Pass: Distributing Gradients
Backpropagation in VFL flows in reverse through the split architecture. Because the label holder computes the final loss $\mathcal{L}(\hat{y}, y)$, it initiates the gradient computation.
- Top Model Update: The label holder computes the loss gradients with respect to the top model parameters $\theta_{top}$ and updates them.
- Gradient Partitioning: The label holder computes the partial derivatives of the loss with respect to the incoming embeddings: $\frac{\partial \mathcal{L}}{\partial \mathbf{h}_A}$ and $\frac{\partial \mathcal{L}}{\partial \mathbf{h}_B}$.
- Transmission: The intermediate gradient $\frac{\partial \mathcal{L}}{\partial \mathbf{h}_A}$ is sent back to Party A.
- Local Chain Rule: Party A uses this intermediate gradient to complete its local chain rule calculation, computing $\frac{\partial \mathcal{L}}{\partial \theta_A} = \frac{\partial \mathcal{L}}{\partial \mathbf{h}_A} \frac{\partial \mathbf{h}_A}{\partial \theta_A}$, and updates its bottom model parameters.
This decoupled update mechanism allows the global model to capture deep, non-linear interactions between disparate features without requiring a centralized data repository. This same principle applies beyond neural networks; for instance, federated tree-based models coordinate feature-level splits across distributed silos by exchanging similarity metrics or cryptographic gradients rather than raw values (Ge et al.; Practical Federated Gradient Boosting Decision Trees).
Real-World Complexities: Hybrid and Semi-Supervised Regimes
While pure vertical partitioning assumes a perfect overlap of samples and a clean split of features, real-world deployments are rarely so tidy.
Data is frequently distributed across both sample and feature axes simultaneously (Overman et al.). For example, in an Internet of Things (IoT) or multi-view sensing network, some edge devices observe the same event from different spatial perspectives (vertical feature splitting), while other devices observe entirely different events elsewhere (horizontal sample splitting) (Liu et al.).
To handle this multi-dimensional fragmentation, hybrid architectures like HoVeFL integrate horizontal and vertical learning strategies simultaneously (Li et al.). The central coordinator aggregates updates from nodes sharing identical feature spaces across different populations, while concurrently managing split-model embedding exchanges for nodes holding complementary features for overlapping individuals.
[ Hybrid Federated Architecture ]
/ \
/ \
(Horizontal Axis) (Vertical Axis)
Aggregate identical Coordinate split-model
features across distinct embeddings across disjoint
sample populations feature spaces
Additionally, managing missing labels or unaligned samples remains a major challenge. When labels are scarce or confined to a single node, vertical semi-supervised learning methods must be deployed to leverage unaligned or unlabeled data across disjoint feature spaces without compromising privacy (Song et al.). By structuring nodes based on feature presence, modern VFL systems balance highly specialized local feature extraction with generalized global aggregation (Seidi et al.).
Key Foundational Work
Federated Machine Learning: Concept and Applications (2019) — contributed 2 concept elements to this direction.
Privacy-Preserving Restricted Boltzmann Machine (2014) — contributed 1 concept elements to this direction.
Distributed Estimation, Information Loss and Exponential Families (2014) — contributed 1 concept elements to this direction.