Better Chinese Sentence Segmentation with Reinforcement Learning

Srivatsan Srinivasan, Chris DyerACL 2021

TL;DR

Chinese sentences often lack clear punctuation, making it hard for machine translation systems to know where one sentence ends and another begins. This paper introduces a reinforcement learning approach to automatically learn how to split Chinese text into segments that lead to better overall translation quality, outperforming existing methods.

What Problem Does This Paper Address?

Machine translation systems typically process text sentence by sentence. However, Chinese, unlike many other languages, doesn't always use clear punctuation like periods to mark sentence boundaries. Commas can be used both within a sentence and to separate complete sentences, creating ambiguity. This ambiguity poses a significant challenge for Chinese-to-English translation systems. They either have to handle very long, potentially multi-sentence inputs, which current models struggle with, or they need a reliable way to segment the Chinese text into appropriate units for translation. Existing methods often rely on heuristics or linguistic rules, which may not align with the actual capabilities of the translation system being used.

Methods

The core innovation of this paper is framing sentence segmentation as a Reinforcement Learning (RL) problem. Here's how it works:

  • Markov Decision Process (MDP): Each Chinese text example is treated as an "episode." The system makes a sequence of decisions (actions) to split or not split at punctuation marks.
  • State: The current state includes information about the input sequence and the history of segmentation decisions made so far.
  • Actions: At each step, the system can either SPLIT at a punctuation mark or CONTINUE without splitting.
  • Reward: The reward is designed to directly optimize the translation quality. Specifically, it's the marginal increase in the BLEU score (a common metric for machine translation quality) achieved by the current segmentation decision, when translating the resulting segments. This reward signal is crucial because it directly links segmentation to the ultimate goal of good translation.
  • Learning Algorithm: The authors use IMPALA, a distributed deep RL algorithm known for its data efficiency and high throughput, to train the segmentation policy. They employ a transformer encoder for state representation and a feed-forward classifier to learn the action distribution.

This RL approach is novel because it learns a segmentation policy that is tailored to the specific translation system being used, rather than relying on general linguistic rules.

Datasets

  • Training and Evaluation: The WMT2020 Chinese-English news translation task dataset is used. This dataset includes aligned paired translations.
  • Baselines: Some baselines (COMMACLASS and HEURISTIC) are trained on the Penn Chinese Treebank dataset.

Key Insights and Ideas

  • Segmentation as a Translation-Enhancing Skill: The paper demonstrates that learning to segment text is not just about linguistic correctness but about optimizing for the downstream task of translation. The RL agent learns to split based on what helps the specific translation model perform better.
  • Adaptability to Translation System Capabilities: The RL approach allows the segmentation policy to adapt to the strengths and weaknesses of the translation system. For instance, if a translation system can handle slightly longer segments well, the RL agent might learn to make fewer splits than a purely rule-based system.
  • Direct Optimization of BLEU: By using BLEU score as the reward signal, the RL agent is directly incentivized to produce segmentations that lead to higher translation quality, which is a more direct and potentially more effective objective than optimizing for intermediate linguistic features.
  • Handling Long Sentences is Crucial: The results highlight that the benefits of this approach are most pronounced for longer sentences, where traditional translation models tend to struggle with truncation.

Key Results

  • Overall Improvement: The proposed RLSEGMENT approach improves the overall BLEU score on the WMT2020 Chinese-English news translation task by 0.3 BLEU compared to a baseline that only splits at unambiguous punctuation.
  • Significant Gains on Long Sentences: For segments containing more than 60 words, RLSEGMENT achieves a substantial improvement of +3 BLEU and a +9+ point improvement in brevity penalty (BP). This indicates that the model effectively mitigates premature truncation issues in long sentences.
  • Competitive Performance: RLSEGMENT performs competitively with supervised baselines (like COMMACLASS) that use syntactic data, but importantly, it does so without requiring any hand-engineered constraints or supervised segmentation data.
  • Example-Based Adaptation: Figure 4 shows an example where RLSEGMENT intelligently chooses to split a sentence into two, rather than three, recognizing that the underlying translation system can handle two-sentence inputs effectively.

Limitations and Future Work

  • Performance Gap to Oracle: While RLSEGMENT outperforms baselines, there's still a gap between its performance and the "oracle" scores (which represent the best possible segmentation). This suggests room for further improvement in segmentation strategies.
  • Scarcity of Long Examples: The authors suspect that the RL learner's sensitivity to BLEU might be amplified by the relative scarcity of long examples in the training data. Supervised learners might ignore these rare cases with less impact on their overall loss function.
  • Inappropriate Segmentations: The paper acknowledges that in some cases, RLSEGMENT's segmentation decisions can lead to worse translations than the baseline, particularly when the segmentation is inappropriate for the specific translation example.
  • Future Work: The authors suggest that this RL-based approach is generic and could be scaled to other languages and translation system variants. Further research into better segmentation strategies is also encouraged.

Summary updated

BibTeX
@inproceedings{srinivasan2021better,
  author = {Srivatsan Srinivasan and Chris Dyer},
  title = {{Better Chinese Sentence Segmentation with Reinforcement Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure