Combining Discourse Markers and Cross-lingual Embeddings for Synonym-Antonym Classification

Michael Roth, Shyam UpadhyayNAACL 2019 · OpenReview

Authors: Michael Roth, Shyam Upadhyay
Venue: NAACL-HLT 2019

Abstract

Standard word embeddings often confuse synonyms with antonyms because both types of words appear in similar contexts. This paper introduces a method to distinguish them in low-resource languages by combining cross-lingual transfers with a simple but powerful linguistic intuition: antonyms frequently co-occur with contrastive discourse markers like "but" or "however."

Introduction

In Natural Language Processing (NLP), distributional semantics—the idea that words with similar meanings appear in similar contexts—is a cornerstone. However, this approach has a "blind spot" regarding antonymy. Because words like "hot" and "cold" are both used to describe temperature or weather, their vector representations (embeddings) end up very close to each other, making it difficult for models to tell them apart. While high-resource languages like English have extensive labeled datasets to help models learn this distinction, most of the world's languages do not. The authors aim to bridge this gap by using monolingual information available in the target language—specifically, how words interact with discourse markers.

Methodology: The LingTrans System

The authors propose LingTrans, a system designed to transfer lexical relation knowledge from a high-resource source (English) to lower-resource target languages.

The LingTrans pipeline: integrating cross-lingual distance and discourse cues for re-ranking translations.

The system consists of three main stages:

  1. Translation Module: Instead of a single best translation, the system identifies the "n-best" translations using vecmap (an unsupervised alignment tool). This prevents the "translation convergence" problem where distinct English words (e.g., "more" and "less") might initially map to the same target word.
  2. Re-ranking via Discourse Markers: To select the most accurate translation pair, the model utilizes contrastive discourse markers (e.g., although, however, but). It calculates Pointwise Mutual Information (PMI) to identify word pairs in the target language that frequently appear near these markers, signaling antonymy.
  3. Classification: These refined translations train a LexNet model—a feed-forward neural network that uses word embeddings to classify the final lexical relation.

Experimental Setup

The approach was evaluated across four languages with varying resources:

  • English: Primary training source (10,932 pairs).
  • German: Development and tuning (811 pairs).
  • Vietnamese: Evaluated using the ViCon dataset (353 pairs).
  • Hindi: Evaluated using a newly crawled dataset from hindi2dictionary.com (1,000 pairs).

Monolingual data for discourse marker analysis was sourced from Wikipedia (German), LORELEI (Vietnamese), and HindEnCorp (Hindi).

Results and Analysis

The LingTrans model significantly outperformed baselines that relied solely on direct translation (BestTrans) or cross-lingual mapping without discourse cues (NoTrans).

F1-score results across German (de), Hindi (hi), and Vietnamese (vi), comparing LingTrans to baseline methods.

As shown in the results table, the inclusion of linguistic features led to substantial gains, particularly in Vietnamese, which saw a ~14-point absolute improvement in F1-score. This suggests that while cross-lingual embeddings provide a foundation, target-language-specific signals are essential for capturing fine-grained semantic nuances. The authors also noted that semantic properties related to antonymy aren't perfectly preserved across language families, making these local signals even more critical.

Conclusion and Future Work

The study demonstrates that discourse markers serve as a robust semantic filter, especially when distributional embeddings fail. However, limitations remain: approximately 30% of errors were attributed to data sparsity for infrequent words. The model also struggled with complex morphological markings and differences in linguistic register. Future work could extend this framework to other lexical relations, such as using markers like "specifically" to identify hypernymy (is-a relationships) in low-resource settings.

Summary updated

BibTeX
@inproceedings{roth2019combining,
  author = {Michael Roth and Shyam Upadhyay},
  title = {{Combining Discourse Markers and Cross-lingual Embeddings for Synonym-Antonym Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=SJW3U7bOZr},
}
Figure