Bridging Disparate Taxonomies via Shared Semantic Embedding Spaces

This direction is merged into another direction: semantic label inconsistency. Its last version stays readable and citable here.

Version 1 of 140 papers49 conceptsfrozen 2026-10-04

In traditional supervised learning, a model’s output is typically a "one-hot" vector. If a dataset has 10 categories, the model learns to map an input to one of 10 orthogonal slots. This works well for isolated tasks, but it creates a "tower of Babel" problem when we try to combine datasets. If Dataset A uses the label "car" and Dataset B uses "automobile," a standard model treats them as entirely unrelated entities.

To build truly universal models—those that can learn from every available source of data—researchers are moving away from discrete integer labels. Instead, they are mapping categories into a shared semantic embedding space, usually anchored by natural language.

The Problem of Taxonomic Conflict

When training on multiple datasets simultaneously, three types of conflicts typically emerge:

  1. Synonyms: Different names for the same concept (e.g., "person" vs. "pedestrian").
  2. Granularity: One dataset might label a "chair," while another labels "armchair" and "stool."
  3. Partial Labeling: A medical dataset might only label the liver, treating the kidneys as "background," while another dataset does the opposite.

In a standard setup, these conflicts lead to "negative transfer," where the model becomes confused and performance degrades. Lingchen Meng and colleagues note in Detection Hub that this often results in a performance drop when datasets are merged. To solve this, the field is shifting toward language-guided classification.

Replacing Indices with Meaning

The core intuition is simple: replace the fixed classification head with a text encoder (like CLIP or BERT). Instead of predicting "Category 5," the model predicts a vector and compares it to the embeddings of label names.

As Yixun Liang demonstrates with MantraNet (Label Name is Mantra), embedding label names into a continuous semantic space allows a model to recognize that "van" and "truck" are closer to each other than "van" and "tree." This proximity allows the model to share features across related categories even if they were never explicitly linked in the training data.

This approach is particularly powerful in domains with scarce data. In 3D representation learning, Xiaoyang Wu introduced Point Prompt Training (PPT) (Towards Large-Scale 3D Representation Learning), which uses language alignment to turn the "negative transfer" of mixed 3D datasets into a performance boost. By mapping inconsistent labels to a shared linguistic manifold, the model treats the union of all datasets as one massive, diverse training set.

Generalizing Across Domains and Tasks

The utility of shared semantic spaces extends far beyond simple object naming:

  • Medical Imaging: In CDPDNet, Jiong Wu uses semantic embeddings to unify partially labeled anatomical datasets. This allows a single model to learn a complete anatomical map by recognizing that "liver" in Dataset A is the same entity as "liver" in Dataset B, even if the other labeled organs differ.
  • Dialogue Systems: Qingyang Wu and James Gung developed DiactTOD (DiactTOD: Learning Generalizable Latent Dialogue Acts), which maps incompatible dialogue act annotations into a shared latent space. This enables systems to generalize to new conversation types without needing specific training for every possible "act" label.
  • Fact-Checking: Isabelle Augenstein has shown that even abstract veracity labels (like "mostly true" vs. "half true") can be mapped into a shared vector space (MultiFC). This allows models to learn from multiple fact-checking organizations that use different rating scales.

Why It Works: The Semantic Anchor

The reason this works is that language acts as a universal regulator. While visual or sensor data varies wildly between datasets, the concepts we use to describe them are relatively stable.

Wei Yin pushed this to the extreme by using Wikipedia descriptions to generate sentence embeddings for class labels (Scaling up Multi-domain Semantic Segmentation). By training on over 2 million images with these embeddings as targets, the model doesn't just learn to recognize a "bus"; it learns the semantic context of what a bus is, enabling zero-shot transfer to entirely new categories.

Similarly, Qilong Zhangli and Dimitris N. Metaxas developed RESI (Resolving Inconsistent Semantics), which uses language embeddings to resolve cases where one dataset labels a "person" and another labels a "face." By treating labels as points in a continuous space rather than discrete buckets, the model can learn that a "face" is a part of a "person," resolving the taxonomic conflict mathematically.

By moving from "what index is this?" to "what does this mean?", these researchers are enabling a new generation of universal models that can ingest any dataset, regardless of how it was originally labeled.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_disparate_taxonomies_via_shared_f710b0de,
  title = {Bridging Disparate Taxonomies via Shared Semantic Embedding Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-disparate-taxonomies-via-shared-semantic-embedding-spaces/rdr_f710b0dec04443c1b21d7be543af16ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_disparate_taxonomies_via_shared_f710b0de_v1,
  title = {Bridging Disparate Taxonomies via Shared Semantic Embedding Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-disparate-taxonomies-via-shared-semantic-embedding-spaces/rdr_f710b0dec04443c1b21d7be543af16ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}