FAUST: Feedback Analysis for User Adaptive Statistical Translation

William Byrne, Lluís MàrquezMTSummit (European Projects) 2013 · OpenReview

Abstract

The FAUST project (2010–2013) aimed to bridge the gap between static machine translation (MT) and real-time user interaction. By leveraging high-volume web traffic from platforms like Reverso.net, the project developed frameworks to capture, filter, and instantaneously incorporate non-professional user feedback into Statistical Machine Translation (SMT) engines. This approach sought to improve translation fluency and accuracy by learning from real-world usage patterns.

Introduction

In the early 2010s, Machine Translation research primarily relied on static, high-quality parallel corpora. While these models were effective for general translation, they often struggled with "fluency" and failed to adapt to the dynamic ways people use language in real-time. The FAUST project addressed two major challenges: establishing a feedback loop for non-professional user data and reducing the latency required to incorporate this feedback into production environments.

Addressing the Feedback Loop and Latency

A significant hurdle in utilizing web-based feedback is its inherent "noise"—casual users may provide incorrect, inconsistent, or even malicious corrections. FAUST focused on building an infrastructure capable of filtering this noise to extract useful signals for model improvement. Furthermore, the project aimed for "instantaneous" adaptation, allowing MT systems to learn from mistakes in near real-time rather than waiting for lengthy retraining cycles.

Methodology

The project combined infrastructure development with novel algorithmic research across several key areas:

  • Experimental Infrastructure: A research-oriented version of Reverso.net (labs.reverso.net) was deployed to log user interactions and test various feedback collection mechanisms.
  • Noise Reduction: Specialized web-oriented mechanisms were developed to filter user-provided feedback, ensuring only high-utility contributions influenced the models.
  • NLG Integration: A technical novelty of FAUST was the integration of Natural Language Generation (NLG)—both deep and shallow—directly into the SMT pipeline. This was designed to improve morphological correctness and overall fluency, moving beyond the "translationese" typical of standard SMT.
  • Asiya Toolkit: The team developed Asiya, an open toolkit for the automatic evaluation of MT systems, featuring a graphical interface and a RESTful web service.
  • Instantaneous Adaptation: Mechanisms were created to inject user corrections back into production engines, enabling the system to "learn" from its errors on the fly.

Data Collection and Annotation

Rather than relying solely on existing datasets, FAUST focused on the creation of novel resources:

  • User Feedback Database: A collection of user interactions and feedback logged from the Reverso.net infrastructure, shared among project partners for modeling interaction.
  • Syntactic Annotation Tools: Tools designed to provide robust syntactic annotations for the informal data typically submitted by web users.
  • Project-Specific Collections: Data acquired specifically to study translation informed by user preferences and corrections.

Key Insights and Results

The project demonstrated that while professional post-editing is the gold standard, the volume of "casual" web feedback offers a unique opportunity for large-scale modeling if noise is managed effectively. The integration of NLG significantly improved morphology and generation in the target language, leading to reduced negative feedback.

FAUST successfully met its objectives within its three-year schedule. Key outcomes included:

  • Commercial Integration: Softissimo integrated FAUST technology into "Reverso Localise," and SDL began planning the use of these mechanisms for professional post-editing environments.
  • Community Tooling: The release of the Asiya toolkit provided a robust platform for MT evaluation.
  • Industry-Academia Synergy: The project successfully transitioned research from institutions like the University of Cambridge and UPC to commercial products.

Limitations and Future Directions

Despite its successes, the project highlighted the ongoing challenge of managing noise in web-based feedback. While the mechanisms developed were effective, web data remains less "trustworthy" than professional editing. Additionally, as the project concluded in 2013, its focus was on Statistical Machine Translation. Adapting these feedback-driven methods to the modern Neural Machine Translation (NMT) paradigm remains a relevant area for future work.

Summary updated

Abstract

William Byrne, Lluis Marquez. Proceedings of Machine Translation Summit XIV: European projects. 2013.
BibTeX
@inproceedings{byrne2013faust,
  author = {William Byrne and Lluís Màrquez},
  title = {{FAUST: Feedback Analysis for User Adaptive Statistical Translation}},
  booktitle = {MTSummit (European Projects)},
  year = {2013},
  url = {https://openreview.net/forum?id=ZGo11g6QmMP},
}
Figure