When training deep learning models for complex visual tasks—such as identifying every object in a busy city street or segmenting every organ in a 3D medical scan—we typically assume access to a "perfect" dataset. In an ideal world, every image is exhaustively labeled with every category or structure present.
In reality, exhaustive annotation is prohibitively expensive and labor-intensive (Hu et al.; Pu et al.). Instead, real-world datasets are frequently partially annotated. A radiologist might meticulously outline the liver for a specific study while ignoring the perfectly visible kidneys and spleen in the same CT scan (Petit et al.). Similarly, a crowdsourced natural image dataset might tag a "cat" but leave "chair" and "window" unannotated (Pourian et al.; Hu et al.).
This creates a severe bottleneck. Standard supervised learning frameworks assume that what is not explicitly labeled as positive must be negative. When applied to partially annotated data, this assumption introduces a destructive bias that cripples model performance.
The False Negative Trap
To understand why partial labels break standard models, consider a multi-label classification network trained with a standard Binary Cross-Entropy (BCE) loss.
If an image contains both a dog and a car, but only the "dog" is labeled, a standard training loop treats the unlabeled "car" as a negative example. When the model correctly detects the car and outputs a high probability for the "car" class, the loss function penalizes it. The model is forced to unlearn valid visual features to satisfy an incomplete ground truth (Elbatel et al.).
Standard Multi-Label Assumption:
[ Labeled: Dog ] ──> Model assumes: (Dog = Present, Car = Absent)
If Model detects Car ──> Penalized! (False Negative Loss)
This phenomenon is especially severe in Single Positive Multi-Label Learning (SPML), where images contain multiple objects but receive only a single confirmed positive tag (Song et al.). In medical segmentation, treating unannotated organs as generic "background" forces the network to suppress its own valid predictions in critical transition zones between anatomical structures (Feng).
To unlock the value of massive, partially labeled datasets, researchers have developed three core strategies: reforming the loss function, blending representations, and aligning distributions.
Strategy 1: Reforming the Loss Function
The most direct way to bypass the false negative trap is to modify how the loss function treats unannotated classes. Instead of assuming missing labels are negative, the loss function can dynamically mask or marginalize them.
Loss Rejection via Information Gaps
One approach is to identify and reject losses that stem from suspected false negatives. For instance, Song et al. introduced the IGNORE method for SPML. By comparing a model's predictions on an original image against a masked version containing only the known positive object, they measure an "information gap." If a label's prediction drops significantly when the known object is isolated, it reveals that the label was likely a hidden positive in the background, signaling the system to reject that specific loss component during training.
Corrective and Marginal Losses
In semantic segmentation, the background class is often a dumping ground for unannotated structures. To fix this, Liu et al. proposed a "Corrective Loss" that remodels background handling, ensuring that unannotated organs are not actively suppressed during incremental learning steps.
Similarly, in distributed or federated learning environments—where different hospitals contribute datasets with completely different annotation protocols—Wang et al. utilized a specialized marginal loss. This formulation ensures that the absence of a label is never treated as definitive evidence of the structure's absence, allowing local models to learn safely from incomplete global signals.
Strategy 2: Representation Blending and Semantic Decoupling
Rather than just changing the loss, another angle focuses on the latent space: how can we manipulate features so that the model inherently generalizes across missing labels?
Semantic-Aware Representation Blending (SARB)
Instead of relying on error-prone pseudo-labels, Pu et al. developed a framework that "blends" features across the dataset. If the model knows what a category looks like in Image A (where it is labeled), it can blend those category-specific features into Image B (where the category's presence is unknown). This allows the model to teach itself using internal data distributions rather than relying on external annotations.
Image A (Labeled: "Cat") ──> Extract Cat Features ┐
├─> Blend ─> Train Model on Image B
Image B (Unlabeled: "Cat?") ──> Latent Space --------┘
Decoupling Vision-Language Models
With the rise of foundational vision-language models like CLIP, partial label learning has shifted toward prompt optimization. However, global image features in these models often become "entangled," making it hard to isolate individual objects when labels are missing.
To solve this, Ruan et al. introduced a semantic decoupling module to separate global features into fine-grained, category-specific representations. Furthermore, Rawlekar et al. demonstrated that the common practice of learning "negative prompts" (e.g., "not a photo of a dog") actually degrades performance when annotations are partial. By replacing negative prompts with simple learned positive embeddings (PositiveCoOp), they achieved state-of-the-art multi-label recognition with significantly fewer parameters.
Strategy 3: Self-Training and Distribution Alignment
A third paradigm relies on the model's own evolving predictions to fill in the blanks, treating the unannotated regions as an unsupervised or semi-supervised learning problem.
Iterative Confidence Relabeling
As a network trains, its predictions on unannotated regions become increasingly accurate. Petit et al. leveraged this by introducing an iterative confidence relabeling loop for organ segmentation, turning unannotated pixels into reliable pseudo-labels over time.
However, pseudo-labeling can introduce confirmation bias if the model becomes confidently wrong. To mitigate this, Liu et al. designed the COSST framework, which pairs self-training with latent-space outlier detection to filter out "noisy" or highly uncertain pseudo-labels before they can corrupt the training process.
Labeled-to-Unlabeled Distribution Alignment
Instead of assigning hard pseudo-labels, we can force the model to treat labeled and unlabeled pixels as coming from the same underlying distribution. Jiang et al. introduced LTUDA (Labeled-to-Unlabeled Distribution Alignment), a framework that aligns feature distributions between fully annotated organs and their unannotated counterparts in other scans. This alignment ensures that the model's feature extractors remain invariant to whether an organ was officially labeled or merely visible.
Application: Stitching Together Heterogeneous Datasets
The ultimate goal of this research direction is to build universal models by stitching together a patchwork of highly specialized, partially labeled datasets.
This is particularly transformative for medical AI. Historically, a clinical team wanting a comprehensive abdominal model had to train separate networks for the liver, kidneys, and spleen because no single dataset labeled all three. Frameworks like DoDNet (Zhang et al.), SegVol (Bai et al.), and jSTABL (Dorent et al.) break this barrier. They allow a single unified backbone to learn simultaneously from disparate datasets—some containing only healthy tissue labels, others containing only specific tumor annotations.
Dataset 1: [Liver Only] ┐
Dataset 2: [Kidney Only] ├─> Unified Framework ─> Universal Multi-Organ Model
Dataset 3: [Spleen Only] ┘
This capability is critical for scaling up decentralized AI. In federated learning, frameworks like FUNAvg (Tölle) allow multiple hospitals to collaboratively train a master model without sharing private data, even when each hospital's local dataset suffers from entirely different annotation gaps.
Large-scale benchmarks such as AMOS (Bai et al.) and AbdomenAtlas-8K (Qu et al.) continue to highlight that mastering partial label dependency is not just an academic exercise—it is the primary path toward training robust, holistic vision systems on real-world data.
Key Foundational Work
Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning (2016) — contributed 1 concept elements to this direction.
Weakly Supervised Graph Based Semantic Segmentation by Learning Communities of Image-Parts (2015) — contributed 1 concept elements to this direction.