Building Salesforce Neural Machine Translation System

Kazuma Hashimoto, Raffaella Buschiazzo, Caiming Xiong, Teresa MarxhallAMTA 2020 · OpenReview

Abstract: Kazuma Hashimoto, Raffaella Buschiazzo, Caiming Xiong, Teresa Marxhall. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track).

2020.

Title slide of the presentation detailing the Salesforce Neural Machine Translation system.

Introduction: The Challenge of Structured Localization

For a global enterprise like Salesforce, localization extends beyond simple linguistic translation; it requires the precise handling of structured data. Salesforce’s online help content is authored in DITA XML, a format featuring over 200 distinct tag types that manage everything from UI references to complex metadata and formatting.

Standard commercial Neural Machine Translation (NMT) systems typically fail in this environment for two primary reasons:

  1. Structural Integrity: Generic models often misplace or "break" XML tags, which disrupts automated publishing pipelines and renders the output technically unusable.
  2. Brand Voice: While commercial models produce grammatically correct general prose, they often fail to capture the specific technical terminology and the unique "Salesforce writing style" required for professional documentation.

To address these issues, Salesforce developed a custom NMT system designed to reduce translation latency and costs while ensuring that international customers receive updates simultaneously with English-speaking users.

Methodology: Customizing the Transformer for XML

The system utilizes a standard Transformer encoder-decoder architecture as its foundation but introduces several critical modifications to handle structured technical text:

  • XML-Tag-Aware Tokenization: Rather than treating tags as standard text strings, the system employs a specialized tokenizer (based on sentencepiece) that recognizes and respects the boundaries of XML tags.
  • Tag Simplification and Mapping: To reduce the complexity of the translation task, intricate tags are mapped to simplified placeholders (e.g., <ph>) during preprocessing, allowing the model to focus on the linguistic context.
  • Copy Mechanisms: This is the core innovation of the system. The model employs a copy mechanism to align tags from the source English text directly to the translated output. By maximizing "copy weights," the system ensures that tags surrounding a specific concept in the source text are correctly applied to the translated equivalent in the target language.
  • Incremental Training: To solve the "cold start" problem associated with new product features, the team incorporates release notes from upcoming versions ($N+1$) into the training data for the current release ($N$). This allows the model to learn new terminology before the primary documentation is even translated.

Data Strategy

The project leveraged a combination of internal proprietary data and public benchmarks:

  • Salesforce Domain Data: Parallel corpora derived from historical releases of Salesforce online help.
  • WMT 2019 Dataset: As part of this research, the authors released a high-quality multilingual dataset for structured documentation translation, now available on GitHub.
  • Release Notes: These were used specifically for fine-tuning the model on the most recent product terminology and feature names.

Key Insights and Discussion

Style Over Substance

A significant finding was that while commercial NMT systems generate high-quality general sentences, the custom Salesforce NMT was far superior at matching the specific Salesforce writing style. In corporate documentation, consistency with the established brand voice is as critical as grammatical accuracy.

Tags as Structural Anchors

Instead of treating XML tags as noise, the system treats them as structural anchors. Utilizing a copy mechanism to carry these anchors from source to target proved more effective than attempting to have the model "learn" the translation of the tags themselves.

Addressing the Terminology Gap

The most persistent challenge remains the translation of brand-new features. The authors found that explicitly injecting release notes into the training pipeline was a necessary tactical "hack" to keep the MT system synchronized with the rapid pace of software development cycles.

Evaluation and Results

The system successfully transitioned from research to production through pilot programs involving Machine Translation Post-Editing (MTPE).

  • Scale: The system currently processes full XML files from English into 16 different languages.
  • Efficiency: By focusing on English strings with low overlap in existing translation memories, the MT system provides the highest value where human translation would be most time-consuming.
  • Qualitative Performance: Human evaluations of 500-string subsets in Japanese, Finnish, German, and French confirmed that the custom NMT outperformed uncustomized commercial alternatives in both structural accuracy and adherence to corporate style.

Future Directions

While the system has significantly improved localization efficiency, human verification remains a vital component of the pipeline. Future research will focus on:

  • Large Pretrained Models: Transitioning from training-from-scratch to fine-tuning massive multilingual models like mBART or XLM-R.
  • Human-in-the-Loop (HITL): Implementing real-time learning from post-editor feedback to refine MT output continuously.
  • Quality Estimation: Developing automated models to identify high-risk segments, allowing human editors to prioritize their efforts on the most complex translations.

Summary updated

BibTeX
@inproceedings{hashimoto2020building,
  author = {Kazuma Hashimoto and Raffaella Buschiazzo and Caiming Xiong and Teresa Marxhall},
  title = {{Building Salesforce Neural Machine Translation System}},
  booktitle = {AMTA},
  year = {2020},
  url = {https://openreview.net/forum?id=XwcYDzs_fL5},
}
Figure