Abstract
In this paper, we compare the relative effects of segment order, segmentation and segment contiguity on the retrieval performance of a translation memory system. We take a selection of both bag-of-words and segment order-sensitive string comparison methods, and run each over both character and word-segmented data, in combination with a range of local segment contiguity models (in the form of N-grams). Over two distinct datasets, we find that indexing according to simple character bigrams produces a retrieval accuracy superior to any of the tested word N-gram models. Further, in their optimum configuration, bag-of-words methods are shown to be equivalent to segment order-sensitive methods in terms of retrieval accuracy, but much faster. We also provide evidence that our findings are scalable.
Introduction
Translation Memory (TM) systems are indispensable tools for professional translators, serving as repositories of previously translated segments. The core task, Translation Retrieval (TR), involves finding the most similar source-language string in the database to suggest a translation for a new input. Historically, research has favored "intelligent" retrieval mechanisms that incorporate linguistic knowledge, such as morphological analysis for word segmentation and order-sensitive algorithms like Edit Distance. However, these methods introduce significant computational overhead. This study empirically evaluates whether such complexity is justified, particularly for Japanese, where word boundaries are not orthographically marked.
Methodology
We evaluated three orthogonal parameters across two Japanese-English datasets: the technical Construction Machinery Corpus and the larger JEIDA Parallel Corpus. The study compared:
- Segmentation Strategy: Raw character-based indexing versus word-based indexing using morphological analyzers (e.g., ChaSen, JUMAN).
- Segment Contiguity: Testing unigrams, bigrams, and mixed (1/2-gram) models.
- Comparison Algorithms: Contrasting bag-of-words (order-oblivious) methods like the Vector Space Model (VSM) and Token Intersection against order-sensitive methods like 3-operation Edit Distance and Weighted Sequential Correspondence (WSC).
Performance was measured using 10-fold semi-stratified cross-validation to determine retrieval accuracy.
Results and Discussion
Accuracy: The Superiority of Character Bigrams
The prevailing assumption that word-based segmentation improves retrieval accuracy is challenged by our findings. Character-based indexing, specifically using bigrams, consistently outperformed word-based models. This is largely due to the error-prone nature of automated morphological analyzers; errors in segmentation propagate through the retrieval pipeline. In contrast, raw characters provide an "error-free" input.
Furthermore, character-based models naturally prioritize technical terminology. In Japanese, foreign loanwords and technical terms are written in Katakana. These words are typically longer than Kanji or Hiragana words, thus generating more character bigrams and receiving higher implicit weighting during retrieval.
Efficiency: Order Sensitivity vs. Bag-of-Words
While order-sensitive algorithms like WSC are intuitively appealing, they are computationally expensive. Our results indicate that for translation retrieval, word order is often redundant. If two strings share a high proportion of character bigrams, they are highly likely to be the same sentence. Consequently, bag-of-words methods like VSM achieve accuracy levels comparable to order-sensitive methods but at a fraction of the cost.
As shown in Figure 1, the retrieval time for order-sensitive methods can be orders of magnitude higher than for VSM. This efficiency gap makes the "dumber" VSM approach the optimal choice for large-scale, real-time TM systems, offering the best balance of speed and consistent accuracy.
Conclusion
This study demonstrates that in translation retrieval, "dumber" is often better. Character bigram indexing provides superior accuracy by avoiding segmentation errors and naturally weighting technical terms. When paired with a bag-of-words algorithm like VSM, the system achieves state-of-the-art accuracy with exceptional computational efficiency. These findings are robust across different datasets and scale effectively, suggesting that simple, low-cost models should be the standard for high-performance translation retrieval.
Summary updated