Label Embedding Approach for Transfer Learning

Rasha Obeidat, Xiaoli Z. Fern, Prasad TadepalliICBO/BioCreative 2016

Introduction

Transfer learning has become a fundamental paradigm in artificial intelligence, enabling models to leverage knowledge from a source task to improve performance on a related target task. However, a significant hurdle arises when the source and target tasks utilize different label sets or ontologies. In such scenarios, standard transfer learning techniques often fail because the output space of the source model does not align with the requirements of the target domain.

In "Label Embedding for Transfer Learning," Obeidat et al. propose a novel framework to bridge this "ontology gap." By utilizing Canonical Correlation Analysis (CCA), the researchers create a shared mathematical space for disparate label sets, allowing for effective inductive transfer learning even when labels are only "intricately related" rather than identical.

Problem Statement: The Ontology Gap

Most domain adaptation techniques assume a constant label space. For instance, a model trained to identify "Person" and "Location" in one dataset is typically expected to identify the same categories in another. However, specialized domains like biomedicine or law often feature variant ontologies. One dataset might use a broad "Protein" tag, while another uses granular tags like "Enzyme" or "Receptor."

Standard methods struggle here because a source model's predictions cannot be directly mapped to the target's schema. The authors address this challenge of inductive transfer learning with variant ontologies, where the goal is to utilize a source task to enhance a target task despite these structural differences in label definitions.

Methodology: The AugmntTr Framework

The proposed approach, referred to as AugmntTr, involves a multi-step pipeline designed to map labels into a continuous latent space:

  1. Inducing Label Embeddings: The core of the method is the use of Canonical Correlation Analysis (CCA). CCA projects both source and target labels into a shared $k$-dimensional space by maximizing the correlation between projected vectors of related labels. This creates a common ground where semantically similar tags (e.g., "Location" in one system and "Geopolitical Entity" in another) are positioned closely together.
  2. Source Model Prediction: A standard sequence labeling model, such as a Conditional Random Field (CRF), is trained on the source domain. This model then generates predictions for the target dataset.
  3. Feature Augmentation: Instead of using raw predicted labels—which can be noisy or rigid—the system retrieves the embedding vector associated with the predicted source label. This vector is appended to the existing feature set of the target instances.
  4. Target Training: A final model is trained on the target dataset using these augmented features, allowing it to benefit from the nuanced relationships captured in the embedding space.

Experimental Setup

The researchers evaluated AugmntTr on Named Entity Recognition (NER) tasks using two distinct datasets:

  • Source Domain: CoNLL 2003, which uses four entity types: Person (PER), Organization (ORG), Location (LOC), and Miscellaneous (MISC).
  • Target Domain: TAC-KBP2015, which uses a more complex six-entity scheme: Person (PER), Title (TTL), Organization (ORG), Geopolitical Entities (GPE), Location (LOC), and Facilities (FAC).

The mismatch between these sets (e.g., CoNLL's "LOC" mapping to both "LOC" and "GPE" in TAC-KBP) represents a classic variant ontology problem.

Results and Discussion

The performance of AugmntTr was compared against two primary baselines: TargetOnly (training solely on target data) and Pred (using raw source predictions as features).

Performance comparison of the AugmntTr method against standard baselines on the TAC-KBP2015 dataset.

As shown in the results, the AugmntTr method significantly outperformed the baselines, achieving a ~9% improvement in F1-score over the standard TargetOnly approach. Interestingly, the Pred baseline performed worse than TargetOnly, suggesting that raw source predictions can introduce harmful noise when ontologies do not align perfectly. In contrast, the label embeddings provided the necessary flexibility for the target model to learn effectively from the source knowledge. This suggests that treating labels as continuous vectors rather than discrete IDs allows models to capture the "degree of relatedness" between different tagging schemes.

Conclusion

By using CCA to discover label relationships automatically, the AugmntTr method provides a scalable solution for transfer learning across variant ontologies. While this preliminary study focused on NER with a small embedding size ($k=5$), it demonstrates the power of label embeddings in bridging disparate data schemas. Future work may explore the scalability of this approach to massive ontologies and its applicability to a wider range of sequence labeling and classification tasks.

Summary updated

BibTeX
@inproceedings{obeidat2016label,
  author = {Rasha Obeidat and Xiaoli Z. Fern and Prasad Tadepalli},
  title = {{Label Embedding Approach for Transfer Learning}},
  booktitle = {ICBO/BioCreative},
  year = {2016},
}
Figure