TL;DR
This paper details the successful automatic compilation of the first large-scale modern Chinese concordances, creating a million-line KWIC index. This work not only provides a crucial resource for understanding modern Chinese but also pioneers methods for computer processing of the language, paving the way for future computational linguistics research.
What Problem Does This Paper Address?
The authors tackle the challenge of deeply understanding modern Chinese, specifically "pai-hua-wen" (written vernacular Chinese). Despite extensive discussion about this language form, there was a lack of detailed statistical and contextual information about its basic vocabulary, syntactic structures, and overall reality. Previous quantitative surveys were limited in scope and used data that might not accurately reflect the language as used by adults. Furthermore, computer applications for Chinese were nascent, with no prior reports of computer-generated Chinese concordances. This project aimed to fill this gap by creating a comprehensive concordance and advancing the methodology for automatic Chinese language processing.
Methods
The core of this project involved the automatic compilation of Key Word In Context (KWIC) concordances. The KWIC style was chosen for its fundamental nature in computer-generated concordances. A key decision was to treat every character, including punctuation, as a "key character" due to the lack of clear word segmentation in Chinese and the significant information content of each character. This approach led to an "All characters in context" index, where each character in the input data generated a line in the concordance. The process involved extensive data input and editing, followed by the use of a modified kanji printer system (T4100) to output the concordances. Two types of indexes were produced: a normal style sorted by the Chinese standard telegraphic code of key characters and a "reverse sorted" index that included preceding characters for more nuanced sorting.
Datasets
The primary dataset used was the "Princeton million-character file," a collection of modern Chinese text compiled over several years. This file contained approximately one million lines of text. The authors selected a diverse range of writings to represent modern Chinese, including novels, philosophical essays, political speeches, and newspaper articles, spanning from the "May 5 Movement" period onwards. Proper nouns were specifically marked to avoid skewing statistical analysis.
Key Insights and Ideas
- Every Character is a Key: The decision to treat every character as a "key word" in the KWIC index, rather than attempting word segmentation (which is difficult in Chinese), was a novel approach that maximized the contextual information captured.
- Bridging the Kanji Gap: The project highlighted the overlap between modern Chinese characters and Japanese Kanji, noting that even with established Kanji sets, a significant number of characters used in modern Chinese were not initially supported by the available kanji printer system. This necessitated the design and incorporation of new fonts.
- Foundation for Future Research: The creation of these large-scale concordances serves as a foundational resource for future linguistic analysis, enabling detailed statistical studies and contextual understanding of modern Chinese.
- Pioneering Chinese NLP: This work represents a significant step in the development of computational linguistics for Chinese, demonstrating the feasibility of automatic processing for complex tasks like concordance generation.
Key Results
The project successfully compiled two sets of one-million-line KWIC indexes for modern Chinese. The data input and editing process was completed, and the concordances were materialized using a modified kanji printer. Statistical analysis revealed that a relatively small number of characters (e.g., 92 characters representing 47% of the data) account for a large portion of the text. The concordances themselves are massive, amounting to over 25,000 pages or 50 volumes.
Limitations and Future Work
The authors acknowledge that the context provided in each concordance line is limited to 55 characters, which might be insufficient for capturing all nuances of meaning. They also note the immense physical size of the printed concordances, suggesting that microfiche versions would have been more practical for portability. The paper concludes by stating that the analysis of these concordances has "just got off the ground," implying that the full potential of this resource is yet to be realized, with further research papers expected to follow.
Summary updated