TL;DR
This paper introduces a novel system for automatically generating abstracts of Chinese and Korean texts and then translating these abstracts into the target language. It combines a sophisticated text indexing method (Natural Hierarchical Network) with corpus-based word segmentation and weighting, followed by an Example-Based Machine Translation (EBMT) system for abstract translation. The system aims to produce accurate summaries and translations, demonstrating promising results for both tasks.
What Problem Does This Paper Address?
The paper tackles the challenge of automatic text summarization and machine translation, specifically for Chinese and Korean languages. Traditional abstracting systems often struggle with accuracy due to reliance on surface clues or complex knowledge bases. Similarly, machine translation systems face difficulties with the nuances of different languages. This work aims to bridge this gap by developing a robust abstracting system that can then leverage an example-based translation approach to produce accurate translations of the generated summaries. The authors highlight the need for systems that can dynamically assess the importance of words and sentences and translate them effectively, especially in the context of Chinese and Korean, which have distinct linguistic structures.
Methods
The proposed system is built upon several key techniques:
- Natural Hierarchical Network (NHN): This is a novel text indexing method that represents text at various hierarchical levels (character, word, sentence, paragraph) using weighted directed graphs. This allows for a rich representation of text structure and content.
- Corpus-Based Word Segmentation: An algorithm is presented that segments text into words based on corpus statistics and a weighting function. This method achieves a high correctness rate (98%) for open tests.
- Word and Sentence Weighting Functions:
- Word Weighting: This function dynamically calculates the importance of words by considering their frequency in context and corpus, as well as word length.
- Sentence Weighting: This function assesses sentence importance by considering the weights of its constituent words, sentence length, and the number of sub-sentences.
- Abstract Generation: The system generates abstracts by selecting sentences with higher weights, maintaining their original order. Experiments show that abstracts produced at 10-20% rates can cover about 90% of important sentences.
- Example-Based Chinese-Korean Machine Translation (EBMT/CK): This system translates the generated Chinese abstracts into Korean. It utilizes a bilingual example corpus, a dictionary, and a character transformation table. Key components include:
- Pattern Sentence and Optimal Cover Set Finding: Identifying relevant example sentences from the corpus.
- Word-Alignment: An approximate matching approach to align words between Chinese and Korean, leveraging linguistic similarities and a bilingual dictionary.
- Target Sentence Generation: Replacing words in the matched pattern sentence with their translated equivalents.
Datasets
The paper mentions the use of:
- Bilingual Example Corpus: Used for training and evaluating the EBMT/CK system.
- Chinese-Korean Dictionary: Used for word alignment.
- Chinese-Korean Character Transformation Table: Used to convert Chinese characters to their Korean pronunciation representations.
- Raw Corpus: Used for word segmentation and weighting. The paper notes that a corpus size of around 50,000 characters is generally sufficient for achieving a realistic and steady coverage rate.
Key Insights and Ideas
- Hierarchical Text Representation is Crucial: The NHN method highlights the importance of representing text at multiple granularities to capture its full structure and meaning for indexing and analysis.
- Dynamic Weighting for Importance: Dynamically calculating word and sentence importance based on context, corpus frequency, and length proves more effective than static methods for abstracting.
- Corpus-Driven Approaches Enhance Accuracy: Leveraging large corpora for word segmentation and weighting significantly improves accuracy compared to dictionary-based or purely rule-based methods.
- Example-Based Translation is Effective for Specific Structures: EBMT is well-suited for translating syntactically uniform structures and handling idiomatic expressions, complementing rule-based approaches.
- Language-Oriented vs. Language-Independent Methods: The abstracting methods are designed to be language-independent, while the translation component is specific to Chinese-Korean.
Key Results
- Word Segmentation: Achieved 98% correctness for open tests.
- Abstract Generation: Abstracts produced at 10-20% rates cover approximately 90% of the important sentences.
- Machine Translation: The EBMT/CK system achieved a translation correctness of over 70% for important words.
- Indexing Speed: The
createNHN(t)algorithm can index at approximately 2000 characters/sec on an IBM PC-486. - Word Segmentation Speed: The running speed is 2000 characters/min for a corpus with 40,000 characters.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while the abstracting methods are language-independent, the EBMT system is specific to Chinese-Korean.
- The storage space for NHN data is noted to be about six times that of the original text data.
- The word alignment method is described as an "approximate match approach," implying potential for further refinement.
- Future Work:
- The authors suggest that the NHN indexing method and corpus-based approaches are meaningful for Multilingual Information Retrieval and Corpus-based Natural Language Processing in general.
- The EBMT/CK system's architecture could potentially be extended to other language pairs.
- The paper mentions an "online learning scheme of translation pairs from the user interaction," which could be a direction for future development.
Summary updated