Weakly Supervised Cross-lingual Semantic Relation Classification via Knowledge Distillation

Yogarshi Vyas, Marine CarpuatEMNLP 2019 · OpenReview

Introduction

Determining the semantic relationship between words is a fundamental task in Natural Language Processing (NLP). While significant progress has been made in monolingual settings—identifying whether "dog" and "animal" share a hypernymy relationship, for instance—extending this to cross-lingual pairs remains a major hurdle. In a multilingual world, translation lexicons are often oversimplified as sets of synonyms. However, semantic coverage rarely aligns perfectly across languages. For example, the English word "father" might map to a Hindi term specifically denoting "father’s brother," representing a relationship of exclusion or hyponymy rather than equivalence.

The primary challenge lies in the lack of labeled cross-lingual datasets. Furthermore, simple "translate-then-classify" approaches fail because translation is inherently ambiguous; a single word can have multiple translations, each potentially carrying a different semantic relationship to the source word. This paper introduces BiLexNet, a framework designed to overcome these obstacles using knowledge distillation and attention mechanisms.

Methodology: The BiLexNet Framework

BiLexNet adapts the monolingual LexNet architecture for cross-lingual tasks. It represents word pairs through a combination of distributional features (bilingual word embeddings) and path-based features (lexico-syntactic paths from dependency trees in parallel corpora).

The BiLexNet architecture utilizes an attention-based distillation process to handle translation ambiguity.

To train the model without labeled cross-lingual data, the authors employ a Teacher-Student distillation approach. An English "teacher" model, trained on high-quality labeled pairs, provides soft labels for the "student" BiLexNet model. The core innovation is an attention mechanism integrated into the distillation process. When translating an English word into a target language (e.g., Hindi), the model encounters multiple dictionary candidates. The attention mechanism weights these candidates, prioritizing translations that are most likely to preserve the semantic relation defined by the teacher, thereby filtering out noise from translation ambiguity.

Experimental Setup and Datasets

The researchers introduced MultiLexRel, a new crowdsourced benchmark containing approximately 1,000 word pairs for both English-Hindi (En-Hi) and English-Chinese (En-Zh). These pairs are annotated with five natural logic relations: Equivalence, Forward Entailment, Reverse Entailment, Exclusion, and Other.

For supervision, the model utilized ~20,000 English pairs from the Lexical-XXXL PPDB. Unlabeled resources included MUSE and MDBG dictionaries, FastText bilingual embeddings, and parallel corpora from WMT18 and the IIT Bombay English-Hindi corpus.

Results and Discussion

BiLexNet demonstrates a significant performance leap over traditional baselines. The model achieved F1 scores of 43.3 for En-Hi and 41.1 for En-Zh, nearly doubling the performance of a random baseline.

Performance comparison of BiLexNet against various baselines on the MultiLexRel dataset.

Crucially, BiLexNet outperformed the "Translation Baseline"—where words are translated to English before classification—by a wide margin. This confirms the authors' insight that translation is a "lossy" process, with 40-45% of semantic relations being altered during simple machine translation. Remarkably, BiLexNet’s performance is within 1–3 F1 points of a fully supervised monolingual English system, despite the student model never seeing a labeled cross-lingual example during training.

Conclusion

BiLexNet provides a robust solution for cross-lingual semantic relation classification by leveraging knowledge distillation and attention. While the model still faces challenges in distinguishing between "Equivalence" and "Exclusion" due to similar distributional contexts, it effectively bridges the gap between languages without the need for expensive manual annotation. Future work may involve reducing dependency on fixed bilingual dictionaries through unsupervised dictionary induction.

Summary updated

BibTeX
@inproceedings{vyas2019weakly,
  author = {Yogarshi Vyas and Marine Carpuat},
  title = {{Weakly Supervised Cross-lingual Semantic Relation Classification via Knowledge Distillation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Hi4mlt3jxvi},
}
Figure