A study of an intelligent algorithm combining semantic environments for the translation of complex English sentences

Ping WangJournal of Intelligent Systems 2022 · DOI

TL;DR

This paper introduces an intelligent pre-processing algorithm designed to solve the "unknown word" problem in Neural Machine Translation (NMT). By leveraging WordNet and semantic context to replace out-of-vocabulary terms with known synonyms before translation, the researchers achieved significant improvements in BLEU scores and translation fluency for complex English-to-Chinese sentences.

What Problem Does This Paper Address?

Despite the rapid advancement of Machine Translation (MT), "complex sentences"—those with intricate grammatical structures or highly specific technical vocabulary—remain a significant hurdle. A primary culprit is the unknown word problem.

When an NMT model encounters a word not present in its training vocabulary, it typically flags it as <UNK>. This leads to incomplete translations, loss of semantic meaning, and broken sentence structures. While sub-word tokenization (like BPE) helps, it doesn't always capture the full semantic intent of rare or technical terms. The author, Ping Wang, seeks to bridge this gap by using "semantic environments" to find intelligent substitutes for these unknown words before the model even begins the translation process.

Methods

The researchers compared two baseline architectures: Recurrent Neural Machine Translation (RNMT) and the Transformer model. To enhance these, they developed an "intelligent algorithm" that functions in five main stages:

  1. Identification: Unknown words in the source sentence are identified.
  2. Candidate Selection: The algorithm queries WordNet to find synonyms with the same part of speech. If no synonyms are found, it looks for hypernyms (broader terms, like "vehicle" for "car") or hyponyms (narrower terms).
  3. Semantic Scoring: To ensure the replacement fits the specific sentence, the algorithm calculates a similarity score based on two factors:
  4. A ternary language model that checks how well the candidate fits with the surrounding words.
  5. A WordNet distance path, measuring how closely related the words are within the semantic network.
  6. Replacement: The unknown word is replaced with the highest-scoring "known" word.
  7. Post-processing: After translation, the algorithm uses a word alignment model (GIZA++) and a bilingual dictionary to ensure the final output correctly reflects the original intent of the replaced word.

Datasets

The study utilized several standard benchmarks for training and evaluation:

  • Training/Development: The Chinese Treebank 6.0 corpus (containing over 20,000 sentences and 647,000+ English words).
  • Evaluation: Multiple NIST (National Institute of Standards and Technology) datasets, specifically NIST 02, 03, 04, 05, 06, and 08. NIST 05 served as the primary development set.

Key Insights and Ideas

  • Contextual Substitution is Key: Simply finding a synonym isn't enough. By using a ternary language model, the algorithm ensures the substitute word makes sense in the specific "semantic environment" of the sentence, preventing awkward or incorrect replacements.
  • Architecture Matters: The experiments reaffirmed the superiority of the Transformer's self-attention mechanism over the RNMT's recurrent approach for complex sentences. The Transformer's ability to handle long-range dependencies makes it naturally better at "complex" syntax.
  • Universal Utility: The intelligent algorithm improved performance for both RNMT and Transformer models. This suggests that semantic pre-processing is a robust strategy that can be layered on top of various NMT architectures to handle edge cases.

Key Results

The quantitative results showed a clear "win" for the combined approach:

  • Transformer Dominance: The baseline Transformer significantly outperformed the RNMT, with an average BLEU score of 42.14 compared to RNMT’s 35.18.
  • The "Boost": Adding the intelligent algorithm raised the Transformer's average BLEU score to 43.7 (an improvement of 1.56 points) and the RNMT's to 37.5 (an improvement of 2.32 points).
  • Error Reduction: The Translation Error Rate (TER) also decreased. For the Transformer combined with the algorithm, the average TER was 57.68.
  • Qualitative Success: In case studies, the system successfully translated previously "unknown" terms like "snow-covered" and "intelligence," resulting in Chinese sentences that were more fluent and closer to the reference translations provided by humans.

Limitations and Future Work

While successful, the author notes a few areas for improvement:

  • Word-Level Focus: The current algorithm only handles word-level replacements. Future research could expand this to phrase-level replacement, which might better capture idiomatic expressions.
  • Low-Frequency Words: The study focused on words entirely missing from the vocabulary. However, "low-frequency" words (those the model has seen only a few times) also cause translation errors and deserve similar semantic treatment.
  • Computational Overhead: The paper does not deeply explore the added latency of querying WordNet and calculating similarity scores during the pre-processing phase, which would be a critical factor for real-time translation applications.

Summary updated

Abstract

Abstract In order to improve the translation quality of complex English sentences, this paper investigated unknown words. First, two baseline models, the recurrent neural machine translation (RNMT) model and the transformer model, were briefly introduced. Then, the unknown words were identified and replaced based on WordNet and the semantic environment and input to the neural machine translation (NMT) model for translation. Finally, experiments were conducted on several National Institute of Standards and Technology (NIST) datasets. It was found that the transformer model significantly outperformed the RNMT model, its average bilingual evaluation understudy (BLEU) value was 42.14, which was 6.96 higher than the RNMT model, and its translation error rate (TER) was also smaller. After combining the intelligent algorithm, the BLEU values of both models improved, and the TER became smaller; the average BLEU value of the transformer model combined with the intelligent algorithm was 43.7, and the average TER was 57.68. The experiment verifies that the transformer model combined with the intelligent algorithm is reliable in translating complex sentences and can obtain higher-quality translation results.
BibTeX
@article{wang2022study,
  author = {Ping Wang},
  title = {{A study of an intelligent algorithm combining semantic environments for the translation of complex English sentences}},
  journal = {Journal of Intelligent Systems},
  year = {2022},
  doi = {10.1515/jisys-2022-0048},
}
Figure