Abstract
As machine learning scales, the reliance on high-quality, human-annotated data has become a significant bottleneck. "Weak supervision" has emerged as a diverse field addressing this challenge, yet it remains fragmented by inconsistent terminology and overlapping methodologies. This paper introduces a formal taxonomy to unify these disparate settings. By decomposing weak supervision into three dimensions—True Label Space, Weak Label Space, and the Weakening Process—the authors provide a rigorous mathematical framework to categorize existing paradigms like semi-supervised, positive-unlabeled, and partial label learning. This unification allows researchers and practitioners to identify common mathematical structures, such as the mixing matrix, across seemingly unrelated domains.
Introduction
The mantra that "data is the new oil" often ignores the reality that refining that oil—labeling data—is expensive, slow, and prone to human error. To circumvent the high cost of ground-truth annotation, the machine learning community has developed various techniques under the umbrella of "weak supervision." However, the field has become a terminological jungle. Terms like "distant supervision," "indirect supervision," and "learning from fuzzy observations" often describe identical underlying mathematical challenges, making it difficult for practitioners to select appropriate algorithms or for researchers to identify truly novel problems.
"The Weak Supervision Landscape" addresses this fragmentation by providing a much-needed map. It moves beyond buzzwords to provide a rigorous taxonomy of how labels can be "weak," offering a common language for the field.
A Three-Dimensional Taxonomy
The core contribution of this work is a framework that decomposes any weak supervision setting into three distinct dimensions. This decomposition allows any dataset or problem setting to be precisely located within the broader landscape.
1. True Label Space
This dimension describes the nature of the underlying ground truth. It defines what the model is ultimately trying to predict. For example, is the target a simple binary choice, a multi-class categorization, or a multi-label setting where an instance can belong to several classes simultaneously?
2. Weak Label Space
This dimension defines the degrees of freedom available to the annotator or the data generation process. It describes the format of the observed data. This might include:
- Soft Labels: Probabilistic assignments rather than hard categories.
- Candidate Sets: A list of possible classes (Partial Label Learning).
- Multiple Annotators: Different, potentially conflicting labels for the same instance.
3. The Weakening Process
Perhaps the most critical dimension, the weakening process describes the transformation from clean ground truth to weak observations. It answers the "how" and "why" of the data's imperfection. Key sub-dimensions include:
- Aggregation: Are labels provided for individual instances or for groups (e.g., Learning from Label Proportions)?
- Class Dependence: Are certain classes inherently more difficult to distinguish, leading to systematic errors (e.g., confusing Huskies with Malamutes)?
- Instance Dependence: Are specific samples particularly confusing due to noise or ambiguity in the image or text itself?
Mathematical Unification via the Mixing Matrix
For non-aggregate settings—where a 1-to-1 relationship between instances and labels still exists—the authors propose a unified formulation using a "mixing matrix" ($T$). This matrix mathematically defines the probability of observing a weak label given a true label: $P(\text{weak} | \text{true})$.
By formalizing the relationship this way, the authors demonstrate that many "different" research areas are actually neighbors. For example, Positive-Unlabeled (PU) Learning is revealed as a specific case of Semi-supervised Learning where the "negative" class is never explicitly labeled, only appearing within an unlabeled pool. Similarly, Superset Learning is defined by a weak label space that is the power set of the true classes, constrained by the requirement that the true label must be contained within the observed set.
Mapping Existing Paradigms
The paper uses these dimensions to categorize well-known machine learning paradigms, providing a clear reference for where they sit in the landscape:
- Semi-supervised Learning (SSL): A mix of true labels and a "null" weak label space.
- Multiple Instance Learning (MIL): An aggregate setting where labels are assigned to "bags" of instances.
- Partial Label Learning (PLL): Where the weak label is a subset of the true label space containing the ground truth.
- Learning from Label Proportions (LLP): An aggregate setting where only the distribution of classes in a group is known.
Key Insights and Future Directions
The framework highlights that understanding how a label becomes weak is as important as the label itself. Distinguishing between instance-dependent noise and class-dependent noise, for instance, fundamentally changes the optimal training strategy.
However, several challenges remain. While the mixing matrix is a powerful theoretical tool, it is often unknown in real-world scenarios, necessitating new methods for matrix estimation. Furthermore, "aggregate" settings like MIL and LLP remain difficult to fit into the same unified mathematical formulation as non-aggregate settings, representing a significant open research frontier. Finally, the authors acknowledge the need to map specific algorithms to these data dimensions, which would allow practitioners to more easily match their specific data "weakness" to the correct computational tool.
Summary updated