BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora

Francis Grégoire, Philippe LanglaisBUCC @ ACL 2017 · OpenReview

TL;DR

This paper explores a deep learning approach to automatically identify parallel sentences in comparable corpora, aiming to reduce reliance on traditional, feature-heavy methods. The authors propose a Bi-GRU based model that leverages pre-trained word embeddings to measure semantic similarity, demonstrating a promising, albeit not state-of-the-art, initial attempt at this challenging task.

What Problem Does This Paper Address?

Finding parallel sentences (sentences that are translations of each other) within comparable corpora (collections of texts on similar topics but not direct translations) is a crucial task for many natural language processing applications, such as machine translation and cross-lingual information retrieval. Traditional methods often rely on complex, hand-engineered features derived from statistical machine translation (SMT) systems. This paper tackles the challenge of identifying these parallel sentences by exploring a more streamlined, end-to-end deep learning framework. The goal is to see if modern neural language models and distributional semantics can achieve this task with minimal preprocessing and feature engineering, offering a more scalable and flexible solution.

Methods

The core of the proposed approach is a bidirectional recurrent neural network with gated recurrent units (Bi-GRU). This model is applied to both the source and target language sentences independently. Each Bi-GRU encodes a sentence in both forward and backward directions, producing a single vector representation that captures the sentence's semantic meaning.

To determine if a pair of sentences is parallel, the model calculates the semantic similarity between their vector representations. This similarity score is then used to estimate the probability that the sentence pair is indeed parallel. A decision threshold is applied to this probability to classify pairs as either parallel or not.

For candidate sentence pair generation, the authors employ an n-best cosine similarity filter. This involves using pre-trained cross-lingual word embeddings to represent words in a shared vector space. Sentence representations are then computed as the normalized sum of their constituent word embeddings. The n-best filter selects candidate pairs based on the cosine similarity between these sentence representations.

Datasets

  • Training Data: The authors used a parallel corpus derived from the Europarl v7 English to French parallel corpus (from WMT'15) for pre-training their models. For their specific training set, they randomly selected 500,000 parallel sentence pairs.
  • Shared Task Data: The paper describes participation in the BUCC 2017 shared task, which involved French and English comparable corpora of approximately 370,000 and 270,000 sentences, respectively.
  • Validation Data: For experimental evaluation, they created validation sets using 9,086 parallel sentences from the shared task training set, augmented with 50 million randomly selected negative sentence pairs.

Key Insights and Ideas

  • Feasibility of Deep Learning: The paper demonstrates that a deep learning framework, relying on distributional semantics and pre-trained embeddings, can be a viable approach for parallel sentence identification, reducing the need for extensive feature engineering.
  • Effectiveness of Bi-GRUs: The Bi-GRU architecture proves capable of generating meaningful sentence representations that capture semantic similarity, which is crucial for the task.
  • Surprising Recall with Simple Filtering: Even with a very basic n-best cosine similarity filter (n=1), the approach managed to capture a significant portion (75.84%) of the parallel sentences found in the shared task training set, highlighting the power of pre-trained embeddings.
  • Precision-Recall Trade-off: Increasing the 'n' value in the n-best filter improves recall but also significantly increases the number of candidate pairs to evaluate, potentially leading to lower precision due to more false positives.
  • Digits Preprocessing Matters: Naively replacing all digits with '0' negatively impacted performance, suggesting that careful handling of numerical information is important for this task.

Key Results

  • The n-best cosine similarity filter, even with n=1, achieved 75.84% recall on the shared task training set.
  • Increasing 'n' to 100 (used naively for the shared task) found 92.32% of parallel sentences but drastically increased the number of candidate pairs.
  • In their model evaluation, using a 40-best filter and a decision threshold of 0.99, they achieved a precision of 12.10%, recall of 70.95%, and F1 score of 20.67%. This low precision was attributed to issues with negative example sampling during training.
  • Experiments on digits preprocessing showed that keeping digits yielded the best performance (Precision: 83.25%, Recall: 65.86%, F1: 73.54%) compared to replacing digits with 0 or removing them.

Limitations and Future Work

  • Low Precision: The primary limitation identified is the significantly low precision score in their final model evaluation. This is attributed to the random sampling of negative examples during training, which led the model to act more as a "quasi-parallel sentences extractor" rather than a precise identifier.
  • Negative Example Sampling: The authors suggest that the random sampling of negative examples is inadequate. Future work should focus on a more effective procedure, such as applying the n-best cosine similarity filter on the training set itself to select more informative negative examples.
  • Post-processing: A post-processing step is also suggested as a potential avenue for improvement.
  • Optimal Threshold: The current approach relies on a fixed, high decision threshold (λ=0.99). A better negative example strategy could allow for a lower threshold, thereby increasing recall.

Summary updated

BibTeX
@inproceedings{gregoire2017bucc,
  author = {Francis Grégoire and Philippe Langlais},
  title = {{BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora}},
  booktitle = {BUCC @ ACL},
  year = {2017},
  url = {https://openreview.net/forum?id=urAr7nvWUy},
}
Figure