Distributed learning paradigms, such as Federated Learning (FL) and Split Learning (SL), are designed on a simple premise: by sharing model updates (gradients) instead of raw data, user privacy can be preserved. However, a growing body of research reveals a critical vulnerability in this setup. Gradients are not abstract, anonymous signals; they are mathematically bound to the training targets. Under close inspection, these shared optimization signals act as a side-channel that leaks private ground-truth labels with startling precision.
The Mathematical Intuition of Gradient Leakage
To understand why labels leak, we must look at how a neural network computes gradients during backpropagation. Consider a standard classification network optimizing a loss function $\mathcal{L}$ (such as cross-entropy) given a model prediction $\hat{y}$ and a ground-truth label $y$.
The gradient of the loss with respect to the weights of the final fully connected layer ($W$) is computed via the chain rule:
$$\frac{\partial \mathcal{L}}{\partial W} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial W}$$
The first term, $\frac{\partial \mathcal{L}}{\partial \hat{y}}$, represents the prediction error vector ($1 - \hat{y}$ for the true class, and $-\hat{y}$ for incorrect classes). Because the ground-truth label $y$ is typically a sparse, one-hot encoded vector, this error term contains an incredibly distinct mathematical signature unique to the true class.
As demonstrated by Hongxu Yin et al. in See Through Gradients: Image Batch Recovery via GradInversion, the gradients of this final fully connected layer act as an immediate giveaway of the ground-truth labels present within a training batch. This classification metadata is highly exposed. As noted by Rui Zhang et al. in A Survey on Gradient Inversion: Attacks, Defenses and Future Directions, an adversary can exploit these final-layer signatures to extract labels with near-perfect accuracy before attempting any complex reconstruction of the underlying input features or images.
Structural Asymmetry in Split and Vertical Learning
The risk of label leakage becomes even more pronounced in frameworks designed around data or model partitioning, such as Split Learning (SL) and Vertical Federated Learning (VFL). In these settings, features or model layers are distributed across different entities. Often, a structural asymmetry arises: one participant holds the target labels (the "label party" or "host"), while other participants hold independent features (the "guest" parties).
This imbalance is a core focus of a comprehensive survey by Mang Ye et al. in Vertical Federated Learning for Effectiveness, Security, Applicability: A Survey. Because the label party must participate in computing the loss, the resulting optimization signals must flow back across the network partition to update the guest models.
Oscar Li et al. demonstrate in Label Leakage and Protection in Two-party Split Learning that the non-label party can easily exploit these backpropagated gradients to steal the private ground-truth labels. A similar vulnerability exists when collaboratively training tree-based models in VFL. As shown by Hangyu Zhu et al. in PIVODL: Privacy-Preserving Vertical Federated Learning Over Distributed Labels, exchanging raw gradients allows honest-but-curious participants to infer private outcomes simply by analyzing the mathematical updates used to determine optimal tree splits.
This structural asymmetry means the label-holding party incurs a disproportionate privacy cost. As observed by Depeng Xu et al. in Achieving Differential Privacy in Vertically Partitioned Multiparty Learning, labels interact with features across all participants, amplifying their sensitivity. Even when distributed configurations attempt to optimize performance—such as the edge frameworks discussed by Guangyu Zhu et al. in Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks or the multimodal systems evaluated by the authors of Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models—the requirement to transmit signals tied to loss calculation introduces severe label inference vulnerabilities.
Why Conventional Defenses Fail
Standard privacy-preserving mechanisms like Differential Privacy (DP), gradient compression, or increasing network depth often fail to block label leakage because label information is remarkably resilient to these perturbations.
For instance, Joohyung Lee et al. show in Exploring the Privacy-Energy Consumption Tradeoff for Split Federated Learning that while pushing the network "cut layer" deeper can successfully mitigate raw data reconstruction attacks, the system remains highly vulnerable to label inference. The underlying categories still leak through the intermediate activations ("smashed data") and gradients passed between layers, a phenomenon also analyzed by Yipeng Li et al. in Convergence Analysis of Split Learning on Non-IID Data.
More alarmingly, sophisticated attacks can bypass active defenses entirely. Junlin Liu et al. in Similarity-Based Label Inference Attack Against Training and Inference of Split Learning demonstrate that similarity-based attacks on intermediate data can reconstruct labels with high accuracy even when defenses like DP-SGD, label differential privacy, or gradient compression are active.
This persistent vulnerability stems from the fact that label spaces themselves act as a fundamental side-channel. As Marlon Tobaben et al. argue in Privacy Leakage via Output Label Space and Differentially Private Continual Learning, applying differential privacy solely to model training parameters fails to protect the output label space, which inherently leaks private data distributions as the label space evolves over time. Thus, as Yunlong Mao et al. note in Secure Split Learning against Property Inference, Data Reconstruction, and Feature Space Hijacking Attacks, feature protection mechanisms do not translate to label protection, necessitating entirely distinct defensive strategies.
Designing Robust Defenses
To counter gradient-based label leakage, researchers are moving away from generic noise injection toward structurally and mathematically tailored defenses.
One approach is to redesign the training architecture to ensure sensitive labels never leave the local device or interact directly with untrusted gradients. For example, Lu Yu et al. in Model Partition and Resource Allocation for Split Learning in Vehicular Edge Networks advocate for U-shaped Split Federated Learning (U-SFL), which keeps labels strictly on-device to prevent central servers from deducing private correlations like location or behavior.
When gradients must be shared, the mathematical relationship between the gradient and the label must be intentionally broken or masked. Oscar Li et al. introduced Marvell in Label Leakage and Protection in Two-party Split Learning, a mathematically optimized noise-injection framework designed specifically to minimize label leakage while preserving model utility.
Alternatively, for complex architectures like Vision Transformers, Seungeun Oh et al. in Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix propose a gradient-splitting mechanism. By isolating the feedback loops so that server gradients are only returned to the specific client that provided the corresponding input patches, they prevent the aggregation of signals that adversaries use to reverse-engineer private ground-truth labels.
Key Foundational Work
Secure Split Learning against Property Inference and Data Reconstruction Attacks (2022) — contributed 2 concept elements to this direction.
Label Leakage and Protection in Two-party Split Learning (2022) — contributed 2 concept elements to this direction.
See Through Gradients: Image Batch Recovery via GradInversion (2021) — contributed 1 concept elements to this direction.