TL;DR
This paper investigates how to best construct bilingual dictionaries for domain-specific cross-lingual embeddings, crucial for transferring NLP capabilities to low-resource languages. They find that using domain-specific words, a moderate dictionary size (around 5k), and prioritizing high-frequency terms leads to the most effective cross-lingual transfer, especially when limited target language data is available.
What Problem Does This Paper Address?
Expanding Natural Language Processing (NLP) models to new languages is a significant challenge, often requiring extensive data annotation. Cross-lingual embeddings offer a solution by allowing knowledge transfer from high-resource to low-resource languages. A common approach involves learning separate monolingual embeddings and then aligning them into a shared space using a "seed dictionary" of translated word pairs. While generic dictionaries and pre-trained embeddings exist, their effectiveness on specialized tasks, particularly in domains with limited parallel data, remains underexplored. This paper aims to fill this gap by identifying best practices for creating domain-specific seed dictionaries to optimize cross-lingual embedding performance for downstream tasks.
Methods
The authors adopt the Multilingual CCA framework, which uses a linear projection to align monolingual embedding spaces into a shared space. This method relies on a seed dictionary, which is easier to obtain than parallel corpora. They experiment with various factors in constructing this bilingual dictionary:
- Source of Data: Comparing dictionaries built from generic sources (like Facebook's MUSE or the Internet Dictionary Project) versus those derived from domain-specific corpora (human resources in this case).
- Frequency of Entries: Investigating the impact of using highly frequent words versus less frequent words from the domain corpus.
- Size of Dictionary: Evaluating dictionaries of different sizes (1k, 5k, and 10k word pairs).
These dictionaries are then used to create cross-lingual embeddings, which are evaluated on a downstream task: sequence labeling for Curriculum Vitae (CV) parsing. The model is trained using English data and tested with projected German/Dutch embeddings, both in a zero-shot setting (no target language training data) and a joint training scenario (with some target language data).
Datasets
- Training Data: The authors use normalized CV data for training monolingual word2vec embeddings.
- English: 4342 annotated documents.
- German: 1947 annotated documents.
-
Dutch: 2383 annotated documents. The data is split into train, development, and test sets with proportions of 70%, 15%, and 15% respectively.
-
Seed Dictionaries:
- Generic: Facebook's MUSE project, The Internet Dictionary Project (IDP).
- Domain-Specific: Top frequent words from the source monolingual corpus (German/Dutch) translated into English using Yandex Translate API.
- Evaluation Data:
- Downstream Task: Curriculum Vitae (CV) parsing, specifically extracting "job title" and "organisation name" from the experience section.
- Intrinsic Evaluation: MUSE test sets consisting of 1,500 translation pairs for precision@1 (P@1) metric.
Key Insights and Ideas
- Domain Matters: Using words from the specific domain for the seed dictionary significantly improves performance, especially in zero-shot scenarios. This highlights that generic word alignments don't capture domain-specific nuances.
- Generic vs. Specialized Resources: Domain-specific dictionaries outperform generic ones, and even specialized generic resources like MUSE are better than general-purpose ones like IDP for this task. This is crucial for industry applications where free, open-source resources might be tempting but less effective.
- Dictionary Size Sweet Spot: A dictionary size of around 5,000 word pairs appears to be sufficient, with diminishing returns and only slight improvements from larger dictionaries (10k). This is important for practical considerations of dictionary creation cost.
- High-Frequency Words are Key: Including high-frequency words, especially those that act as section headers or common context words for entities, is crucial for zero-shot performance. These words help anchor the embedding space.
- Intrinsic vs. Extrinsic Evaluation: The intrinsic P@1 metric on translation pairs doesn't always correlate with downstream task performance, underscoring the importance of evaluating cross-lingual embeddings on the actual task.
Key Results
- Task Performance: Using domain-specific dictionaries with 5k high-frequency words achieved the best results for both joint training and zero-shot scenarios. For example, in the DE-EN zero-shot setting, performance improved from 61.5% (IDP + 5k) to 75.8% (domain + 5k + high freq).
- Impact of Target Data: The gain from knowledge transfer (cross-lingual gain) is more pronounced when less target language data is available. With a strong English-only model, using cross-lingual embeddings achieved comparable results to a model trained on only 15% of the low-resource language data.
- Intrinsic Metric Discrepancy: The P@1 metric ranked MUSE and IDP dictionaries in reverse order compared to their performance on the CV parsing task, highlighting its limitations for evaluating task-specific effectiveness.
Limitations and Future Work
- Limitations: The study is limited to German and Dutch as low-resource languages and English as the high-resource language. The authors acknowledge that the impact of increasing training data size diminishes as more data becomes available.
- Future Work: The authors plan to extend their research to other language pairs, including Slavic languages and more distant pairs like English-Russian. They also aim to explore cross-lingual subword embeddings, which are particularly relevant for languages with complex morphology.
Summary updated