Improving Multilingual Summarization: Using Redundancy in the Input to Correct MT errors

Advaith Siddharthan, Kathleen R. McKeownEMNLP 2005 · OpenReview

Abstract

In this paper, we use the information redundancy in multilingual input to correct errors in machine translation and thus improve the quality of multilingual summaries. We consider the case of multi-document summarization, where the input documents are in Arabic, and the output summary is in English. Typically, information that makes it to a summary appears in many different lexical-syntactic forms in the input documents. Further, the use of multiple machine translation systems provides yet more redundancy, yielding different ways to realize that information in English. We demonstrate how errors in the machine translations of the input Arabic documents can be corrected by identifying and generating from such redundancy, focusing on noun phrases.

Introduction

In the mid-2000s, machine translation (MT)—particularly for language pairs like Arabic and English—was notoriously noisy. Systems frequently produced ungrammatical sentences and inaccurate word choices. For researchers building multilingual summarizers, these errors were inherited by the final output, significantly degrading readability and trustworthiness.

This paper identifies a unique opportunity within multi-document summarization: redundancy. When multiple news reports cover the same event, they inevitably refer to the same entities and facts repeatedly. While a single MT system might fail on a specific reference, the aggregate of multiple mentions across different documents and different MT systems often contains the correct information. The authors propose a system to systematically harvest this redundancy to "repair" English noun phrases (NPs) in the generated summaries.

Methodology

The authors developed two distinct strategies for cleaning up noun phrases, depending on whether the entity is a person or an arbitrary object/concept.

1. Generating References to People

For references to individuals, the system employs a formal generation approach. It first performs semantic tagging using tools like BBN’s IdentiFinder and WordNet to extract attributes such as name, role, organization, and country. These attributes are aggregated into an Attribute Value Matrix (AVM).

Attribute Value Matrices (AVMs) used to aggregate entity information across multiple documents and translations.

By performing coreference resolution and frequency-based filtering, the system removes MT "hallucinations." Finally, it uses syntactic frames learned from the Reuters corpus (e.g., [Country] [Role] [Person]) to generate a clean, grammatical reference, such as "Iraqi United Nations representative Nizar Hamdoon," rather than relying on a single potentially garbled MT string.

2. Selecting Arbitrary Noun Phrases

For general NPs where formal generation is too complex, the system uses a statistical selection approach. It aligns NPs across different translations of the same sentence using the BLAST algorithm and clusters similar NPs across documents using SimFinder. To select the most fluent version from a cluster, the system calculates the Pointwise Mutual Information (PMI) of word pairs. This logic assumes that words naturally occurring together in English (collocations) are more likely to be part of a correct translation than a noisy MT error.

Experimental Results

The system was evaluated using the DUC 2004 Multilingual summarization task, which provided Arabic news reports translated into English by two different MT systems (ISI and IBM). The generated summaries were compared against human-written references.

F-Measure performance for Unigrams and Bigrams relative to input redundancy. The "Generated" line shows significant improvement as redundancy increases.

A key finding was the "5-Reference Rule": the quality of generated references improves significantly once an entity is mentioned at least five times in the input. As shown in the performance graphs, the F-measure for the generated system consistently outperforms baselines, particularly as redundancy increases. This is ideal for summarization, as the most frequently mentioned entities are typically the most important for the summary.

Quantitatively, the system achieved a unigram precision of 0.847, significantly higher than the 0.753 achieved by the best baseline. It also showed dramatic improvements in standard MT metrics, with a BLEU score of 0.898 compared to the baseline's 0.499. In blind human evaluations, judges rated the rewritten sentences higher for both fluency and accuracy.

Conclusion and Future Work

By leveraging the "wisdom of the crowd" inherent in redundant multilingual inputs, the authors successfully mitigated the noise of 2005-era machine translation. While the current work focuses on noun phrases, the results demonstrate that redundancy is a powerful filter for stochastic MT errors. Future research aims to extend this redundancy-based correction to verb groups and argument structures to move toward a fully generative, noise-robust summarization framework.

Summary updated

Abstract

In this paper, we use the information redundancy in multilingual input to correct errors in machine translation and thus improve the quality of multilingual summaries. We consider the case of multi-document summarization, where the input documents are in Arabic, and the output summary is in English. Typically, information that makes it to a summary appears in many different lexical-syntactic forms in the input documents. Further, the use of multiple machine translation systems provides yet more redundancy, yielding different ways to realize that information in English. We demonstrate how errors in the machine translations of the input Arabic documents can be corrected by identifying and generating from such redundancy, focusing on noun phrases.
BibTeX
@inproceedings{siddharthan2005improving,
  author = {Advaith Siddharthan and Kathleen R. McKeown},
  title = {{Improving Multilingual Summarization: Using Redundancy in the Input to Correct MT errors}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  url = {https://openreview.net/forum?id=SybE0MWuZr},
}
Figure