TL;DR
This paper tackles the challenge of translating Signed Japanese (SJ) into spoken Japanese, a low-resource task due to the inherent differences between signed and spoken languages. They propose a novel pipeline approach that first uses statistical machine translation to map sign language glosses to Japanese words, then iteratively refines the translation by adding missing post-positional particles and conjugating verbs using transformer-based models. This method significantly outperforms a baseline SMT approach, achieving substantial improvements in BLEU scores.
What Problem Does This Paper Address?
Translating sign language into spoken language is a complex task, especially for low-resource languages like Signed Japanese (SJ). A major hurdle is the difference in linguistic structure. Unlike spoken Japanese, SJ often omits function words like post-positional particles and lacks explicit morphological inflection for verbs, adjectives, and auxiliary verbs. Current translation systems typically rely on large parallel corpora, which are scarce for SJ. This paper addresses this gap by proposing a pipeline method that specifically targets these linguistic differences, aiming to produce grammatically correct and fluent Japanese translations from SJ glosses, even with limited data.
Methods
The authors propose a six-step pipeline translation method.
- Gloss to Japanese Phrase Translation (Step 0): A phrase-based statistical machine translation (PBSMT) model (Moses) is used to translate the SJ gloss sequence (G) into an initial Japanese sequence (S_PP-C). This step focuses on word-level translation without considering particles or conjugations.
- Post-positional Particle Estimation (Step 1): A transformer-based seq2seq model (s2s_m1) is employed to estimate and insert missing post-positional particles, converting S_PP-C to S+PP-C.
- Verb/Adjective/Auxiliary Conjugation (Step 2): Another transformer-based seq2seq model (s2s_m2) estimates the appropriate morphological inflections (conjugations) for verbs, adjectives, and auxiliary verbs, transforming S+PP-C to S+PP+C.
- Particle Removal (Step 3): Previously added post-positional particles are removed from S+PP+C to obtain S-PP+C.
- Particle Re-estimation (Step 4): A third transformer-based seq2seq model (s2s_m3) re-estimates and inserts post-positional particles into S-PP+C, resulting in S+PP+C. This step aims to correct potential errors from the initial particle estimation by considering the conjugated forms.
- Canonical Form Conversion (Step 5): The conjugated words in S+PP+C are converted back to their canonical forms, yielding S+PP-C.
Steps 3, 4, 5, and 2 are then iteratively repeated until the translation output converges or a maximum number of iterations (10) is reached. This iterative refinement is a key aspect of their approach.
Datasets
- In-house SJ and Japanese Parallel Corpus: This corpus contains 1,086 sentence pairs with over 7.5K glosses and over 11K Japanese words. The sentences are derived from spontaneous conversations, likely from municipal offices, covering topics like resident registration, pensions, and unemployment insurance.
- Out-of-domain Japanese Monolingual Corpus: A subset of the Balanced Corpus of Contemporary Written Japanese (BCCWJ) was used. This corpus, containing over 195K sentences and 3.9M Japanese words, was utilized to generate pseudo-parallel corpora for training the transformer-based seq2seq models.
Key Insights and Ideas
- Leveraging Linguistic Differences: The core idea is to explicitly address the grammatical differences between SJ and spoken Japanese (missing particles, lack of conjugation) rather than ignoring them.
- Iterative Refinement: The pipeline's iterative nature, where steps are repeated, is crucial for improving translation quality. This allows the system to refine its estimates of particles and conjugations over multiple passes.
- Pipeline vs. End-to-End: The paper demonstrates that a carefully designed pipeline approach can outperform end-to-end models, especially in low-resource settings, by breaking down the complex translation task into manageable sub-problems.
- Importance of Post-positional Particles: The results suggest that estimating post-positional particles is more challenging and uncertain than estimating verb conjugations, as indicated by the BLEU scores of individual seq2seq models.
Key Results
The proposed pipeline method, SMT+Iterative_s2s, significantly outperformed the SMT baseline.
- BLEU-3/4 Improvement: The method achieved +4.4/+4.9 points higher BLEU-3/4 scores compared to the SMT baseline.
- Performance of Pipeline Models: All pipeline models combining SMT and seq2seq models outperformed models that directly translated gloss to Japanese (e.g., naive, LSTM, G2T).
- Iterative Convergence: Approximately 72% of the results converged within the first iteration, and 26% converged within the second, indicating the efficiency of the iterative process.
- Individual Model Performance: The seq2seq model for conjugation estimation (s2s_m2) achieved a very high BLEU4 score (98.75), while particle estimation models (s2s_m1, s2s_m3) had lower scores (74.20, 75.06), highlighting the difficulty of particle estimation.
Limitations and Future Work
- Limited Corpus: The in-house parallel corpus is relatively small, which is a characteristic of low-resource settings.
- Lack of Non-Manual Signals (NMSs): The current parallel corpus does not include NMSs (facial expressions, head movements, etc.), which are crucial grammatical markers in sign language. The authors suggest that incorporating visual features from signs could improve translation quality.
- Domain Mismatch: Using an out-of-domain Japanese corpus for training seq2seq models led to a slight degradation in the SMT baseline's performance, suggesting potential domain mismatch issues.
- Gloss Ambiguity: The inherent polysemy of gloss words and the information bottleneck in gloss notation are identified as causes of ambiguity.
- Future Work: The authors are interested in investigating how incorporating visual features from signs would improve their translation method. They also suggest that their iterative, multi-model pipeline approach could be adapted for similar grammatical error correction tasks.
Summary updated