Unifying Heterogeneous Label Spaces Across Datasets

478 concepts304 papers1 proposal

Modern machine learning thrives on scale. The standard recipe for building more robust models is simple: aggregate more data. However, when we attempt to merge multiple datasets within the same domain, we immediately run into a fundamental structural barrier. Machine learning models typically operate under a closed-world assumption, meaning they expect a fixed, predefined, and mutually exclusive set of semantic labels.

In reality, different datasets are curated by different groups, at different times, with different objectives. This results in heterogeneous label spaces—a phenomenon where the underlying taxonomy of categories is fragmented, inconsistent, or outright contradictory across data sources. Naively combining these datasets causes standard training algorithms to fail, as the model receives conflicting supervisory signals.

To build truly universal models, we must move away from rigid, dataset-specific taxonomies and design systems capable of dynamically reconciling semantic label inconsistencies.


The Dimensions of Taxonomic Friction

Semantic label inconsistency is not a single problem, but rather a spectrum of misalignments that occur when different human annotators attempt to categorize the world. We can break this friction down into three core types:

1. Granularity Mismatches (Hierarchical Inconsistency)

Different datasets often look at the same object through lenses of varying detail. As noted by the authors of MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation, one dataset might use a broad aggregate category like person, while another distinguishes between fine-grained sub-categories like pedestrian and rider. Similarly, in medical imaging, Kemnitz et al. observe in Combining Heterogeneously Labeled Datasets For Training Segmentation Networks that one clinical study might label an entire anatomical structure, while another isolates its specific sub-components.

When forced into a standard flat classification layer, these hierarchical relationships clash. A rider from Dataset A is technically a person, but if the model predicts person on Dataset B's images, it is penalized for not predicting the more specific label.

2. Naming Conventions and Synonyms

Identical semantic concepts are frequently masked by different vocabulary. In animal pose estimation, Ye et al. point out in SuperAnimal pretrained pose estimation models for behavioral analysis that fragmented research groups use entirely inconsistent naming conventions for identical anatomical keypoints. This vocabulary gap spans domains: Hu et al. document how mismatched activity labels in smart-home datasets hinder transfer learning in Transfer Learning for Activity Recognition via Sensor Mapping, while Dan Wu demonstrates a stark lack of taxonomic alignment between professional and community-curated Q&A platforms in Comparing IPL2 and Yahoo! Answers: A Case Study of Digital Reference and Community Based Question Answering.

3. Conceptual Divergence and Subjective Boundaries

The most challenging form of inconsistency occurs when the definition of a category itself changes based on the context of the dataset. For instance, Steffen Eger highlights in Multi-Task Learning for Argumentation Mining in Low-Resource Settings that fundamental categories like "claims" and "premises" vary wildly across text corpora.

In content moderation, Ruiter et al. show in LSV-UdS at HASOC 2019: The Problem of Defining Hate that what constitutes "toxic content" or "hate speech" shifts across platforms, making fine-grained data fusion highly problematic. Even in geometric tasks, Wang et al. note in Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection that different 3D datasets enforce distinct annotation rules for bounding box boundaries around the exact same object classes.


Why Naive Merging Fails: The Gradient Contradiction

To understand why label inconsistency paralyzes standard neural networks, consider what happens to a model's optimization landscape during joint training.

Suppose we train a semantic segmentation network on two datasets: Dataset $A$ (an urban driving dataset) and Dataset $B$ (a hazard-detection dataset). In Dataset $A$, a small brick lying on the road is unannotated and lumped into the broad background class road. In Dataset $B$, which focuses on anomalies, that exact same visual feature is explicitly labeled as obstacle.

Dataset A: [Visual Feature: Brick] ----> Labeled as "Road"
Dataset B: [Visual Feature: Brick] ----> Labeled as "Obstacle"

When the network processes an image from Dataset $A$ containing a brick, a standard cross-entropy loss forces the model to suppress its probability for obstacle and increase it for road. Conversely, when it encounters a similar image from Dataset $B$, the loss forces it to do the exact opposite.

As Sun et al. demonstrate in Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images, these conflicting gradients cancel each other out. The network suffers from persistent semantic confusion, preventing it from converging on a distinct representation for either class. Instead of learning from the collective volume of both datasets, the model's performance degrades on both.


Mechanisms for Label Space Harmonization

To overcome this, researchers have developed several distinct paradigms to map disparate label spaces into a unified, coherent representation.

Language-Guided Taxonomy Unification

Instead of treating labels as discrete, orthogonal one-hot vectors (where cat and tabby are mathematically as distinct as cat and airplane), modern frameworks leverage the continuous semantic space of pre-trained vision-language models like CLIP.

Zhou et al. pioneer this in LMSeg: Language-guided Multi-dataset Segmentation by replacing rigid classification heads with text embeddings. By projecting text descriptions of labels into a shared embedding space, the model inherently understands hierarchical and synonymous relationships.

One-Hot Space (Rigid):
[Cat]     = [1, 0, 0]
[Tabby]   = [0, 1, 0]  --> Distance = Max

Language Embedding Space (Fluid):
Embedding("Cat")   • Embedding("Tabby")   --> High Similarity
Embedding("Cat")   • Embedding("Airport") --> Low Similarity

This language-guided approach allows models like ScaleDet, developed by Chen et al., to dynamically reconcile overlapping categories across massive object detection datasets without requiring manual relabeling.

Visual and Data-Driven Alignment

When language descriptions are insufficient or ambiguous, the visual data itself can be used to discover how label spaces overlap. Zhou et al. introduce a framework in Simple Multi-dataset Detection that automatically unifies inconsistent taxonomies by treating the true unified label as a latent variable. By observing which visual features are shared across datasets, the model learns a probabilistic mapping between the fragmented taxonomies, allowing a single detector to perform seamlessly across distinct benchmarks.

Partial and Super-Label Loss Formulations

Another elegant mathematical solution is to reformulate the loss function to tolerate partial or ambiguous supervision. In Combining Heterogeneously Labeled Datasets For Training Segmentation Networks, Kemnitz et al. introduce the Super Label Aware Crossentropy ($C^{slac}$) loss.

Instead of penalizing a model when its prediction doesn't perfectly match a dataset's specific label, the loss function groups fine-grained and coarse labels into "super-labels." If a model predicts pedestrian on an image labeled only as person, the $C^{slac}$ loss recognizes that pedestrian is a valid subset of person and relaxes the penalty. This prevents the gradient contradictions that typically ruin multi-dataset training.

Similarly, Kumar et al. address this in Learning specific-class segmentation from diverse data by treating missing or ambiguous taxonomic information as latent variables within a self-paced learning framework, ensuring that the model only updates its weights when the semantic signal is clear.


The Path to Universal Generalization

Resolving semantic label inconsistency is not just a data-cleaning exercise; it is a prerequisite for building generalizable foundation models. When label spaces are successfully unified, models exhibit a dramatic leap in zero-shot generalization to unseen environments.

By breaking down the artificial walls between dataset-specific taxonomies, frameworks like UniDet3D (Kolodiazhnyi et al. in UniDet3D: Multi-dataset Indoor 3D Object Detection) and OneFormer (Jain et al. in OneFormer: One Transformer to Rule Universal Image Segmentation) prove that architectural complexity can be stripped away when data diversity is unlocked. Resolving semantic conflicts allows networks to stop memorizing dataset-specific annotation quirks and start learning the true, underlying invariant structure of the world.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 326 papers
  2. Version 1 · 2026-10-04 · 490 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_label_inconsistency_267719d5,
  title = {semantic label inconsistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-label-inconsistency/rdr_267719d5533e4cfcac8db99a87451367},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_semantic_label_inconsistency_267719d5_v2,
  title = {semantic label inconsistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-label-inconsistency/rdr_267719d5533e4cfcac8db99a87451367/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake