Learning Better Representations Using Auxiliary Knowledge

Saed RezayiAAAI 2023 · DOI

Abstract

Representation Learning (RL) is essential for modern AI, yet standard models often face limitations due to data sparsity and a lack of domain-specific context. This paper investigates the systematic integration of "auxiliary knowledge"—including external text, knowledge graphs, and cross-domain datasets—to enhance representation quality. We present several novel frameworks: the "Edge" framework for soft graph augmentation, AgriBERT for specialized agricultural language modeling, and XDC for cross-domain clustering. Experimental results show that these methods significantly outperform vanilla models, with AgriBERT achieving a 20% boost in precision. Furthermore, we propose using external knowledge as a defensive anchor against adversarial attacks on knowledge graphs.

Introduction

Representation Learning transforms complex data into low-dimensional vectors, enabling machines to process information efficiently. However, "vanilla" models often hit a performance ceiling. Knowledge graph embeddings frequently suffer from missing links, while language models are restricted to their training vocabulary. In niche domains, these models struggle to capture subtle relationships. This research addresses the challenge of filling these "knowledge gaps" by integrating external, unstructured, or cross-domain data into the learning process.

Methodology

The research encompasses a multi-pronged approach to knowledge integration:

The "Edge" Framework (Text-to-Graph)

To mitigate sparsity in knowledge graphs, we propose a "soft" augmentation method. Instead of adding hard links based on simple co-occurrence, we generate a rich, augmented graph using external text (e.g., WordNet). A graph alignment term in a shared embedding space is then used to distill relevant information and suppress noise.

AgriBERT (KG-to-Text)

In the agricultural domain, where data is scarce, we combined a transformer-based language model with a domain-specific knowledge graph (FoodOn). The model was fine-tuned on a corpus of 46,446 agricultural papers containing over 300 million tokens.

Cross-Domain Clustering (XDC)

For clustering short, sparse text, we developed an adversarial learning framework. XDC utilizes a labeled "source" domain to help model the underlying distribution of an unlabeled "target" domain, effectively addressing the "distribution shift" between different text types.

Robustness via External Knowledge

We propose defending knowledge graphs against adversarial attacks by anchoring entity representations to their textual descriptions. This involves a transformation matrix ($M$) that projects word embeddings into the graph embedding space, creating a linguistic "ground truth" that is harder for adversaries to manipulate.

Key Results and Insights

  • AgriBERT Performance: Enhancing a language model with the FoodOn knowledge graph boosted precision@1 by 20% compared to standard fine-tuned models.
  • Soft vs. Hard Augmentation: Our findings indicate that "soft" augmentation—using alignment layers to filter relevant data—is more effective than simply adding raw external data, which often introduces excessive noise.
  • Knowledge as a Shield: Tying graph entities to their linguistic definitions provides a security layer, making it significantly more difficult for an adversary to "trick" the model by altering graph edges.

Future Work and Research Roadmap

Current methods for enhancing robustness are being refined to target the local structure of graphs most affected by attacks. The project is currently moving into its final phases of evaluation and thesis documentation.

Research project timeline for implementation, evaluation, and submission.

As shown in the timeline above, the coming months will focus on finalizing evaluations and submitting results to major conferences such as NeurIPS or AAAI.

Summary updated

BibTeX
@inproceedings{rezayi2023learning,
  author = {Saed Rezayi},
  title = {{Learning Better Representations Using Auxiliary Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26927},
}
Figure