Unifying Heterogeneous Label Spaces for Universal Perception

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2490 papers603 conceptsfrozen 2026-10-04

In computer vision, the "one dataset, one model" paradigm has long been a bottleneck. While we have massive amounts of labeled data across datasets like COCO, LVIS, and Waymo, we cannot simply "add" them together. The primary obstacle is not the pixels or the point clouds, but the semantic misalignment of their taxonomies.

One dataset might label a "Van" and a "Truck" separately, while another groups them under "Vehicle." A third might treat a person inside a car as part of the "Car" class, while a fourth labels them as a "Pedestrian." This lack of a unified label space creates conflicting supervision signals, where the same visual features are mapped to different categorical "truths," leading to what researchers call "dataset interference."

Recent work focuses on bridging these annotation gaps to create universal models that leverage the collective intelligence of diverse data sources.

The Taxonomy Conflict

The challenge of multi-dataset training is fundamentally a problem of logic and language. As Marin Oršić and colleagues point out in Multi-domain semantic segmentation with pyramidal fusion, even mathematical mapping often fails to resolve fundamental conflicts in labeling protocols—such as whether to label an occluded object or the surface occluding it.

These conflicts manifest in three main ways:

  1. Granularity Mismatches: Likun Cai and the BigDetection team note that disparate taxonomies prevent direct aggregation because one source's "Vehicle" is another's "Car," "Truck," or "Bus."
  2. Semantic Overlap: Identical physical objects are assigned different names across domains (e.g., "Pedestrian" vs. "Person").
  3. Foreground-Background Contradictions: Lingchen Meng observes in Detection Hub that an object category labeled as foreground in one dataset might be ignored as background in another, confusing the model's gradient updates.

Strategy 1: Mapping to a Unified Taxonomy

The most direct solution is to construct a "super-taxonomy." In UniDet3D, Maksim Kolodiazhnyi and his team unified six major indoor 3D datasets by manually reconciling their label spaces. By stripping away dataset-specific complexities, they proved that data diversity alone can drive state-of-the-art results.

Similarly, the BigDetection framework unifies LVIS, OpenImages, and Objects365 into a single "super-dataset" of 600 categories. This approach relies on "label-aligned transfer" (LAT) to ensure that when a model sees a "Truck" in one dataset, it understands it is the same concept as a "Vehicle" in another.

Strategy 2: Language-Guided Alignment

A more scalable approach moves away from fixed integer labels (e.g., Class 1, Class 2) and toward language embeddings. By treating labels as text, models can use the inherent semantic relationships in language to bridge gaps.

MantraNet replaces traditional fixed-label decoders with a language-guided approach. By embedding label names into a continuous semantic space using pre-trained language models, the system can train on heterogeneous datasets simultaneously. This allows the model to recognize that "Car" and "Automobile" are semantically adjacent, even if they appear in different datasets.

Xin Wen extends this in Point Prompt Training (PPT), using "domain prompts" and language alignment to turn the "negative transfer" of mixed data into a performance boost.

Strategy 3: Hierarchical and Dataset-Aware Architectures

Instead of forcing all labels into a flat list, some researchers model the hierarchical nature of the world. Ali Farhadi pioneered this with YOLO9000, using a hierarchical tree to relate broad classification labels to specific detection boxes.

More recently, Yash Jain introduced DAMEX, which uses a Mixture-of-Experts (MoE) architecture. During training, the model routes data to specialized "experts" based on the source dataset. This allows the model to learn dataset-specific nuances (like specific camera distortions or labeling quirks) while sharing a backbone that understands universal visual features. Point-MoE applies a similar logic to 3D point clouds, allowing for cross-domain generalization without requiring dataset labels at inference time.

Why It Works: From Silos to Synergy

The intuition behind this research direction is that semantic consistency is a prerequisite for data scaling. When we resolve label conflicts, we don't just get more data; we get more diverse data.

As demonstrated by Xingyi Zhou in Simple Multi-dataset Detection, a model trained on a unified taxonomy generalizes significantly better to unseen environments than a model trained on any single dataset. By bridging the annotation gap, we move from specialized "island" models toward universal perception systems that understand the visual world as a cohesive whole.

Papers in this version

Showing 60 of 490 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 326 papers
  2. Version 1 · 2026-10-04 · 490 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_label_inconsistency_267719d5,
  title = {semantic label inconsistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-label-inconsistency/rdr_267719d5533e4cfcac8db99a87451367},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unifying_heterogeneous_label_spaces_for__267719d5_v1,
  title = {Unifying Heterogeneous Label Spaces for Universal Perception},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unifying-heterogeneous-label-spaces-for-universal-perception/rdr_267719d5533e4cfcac8db99a87451367/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}