A Chinese Corpus for Linguistic Research

Chu-Ren Huang, Keh-jiann ChenCOLING 1992

TL;DR

This paper introduces the construction of a comprehensive corpus for both modern and classical Chinese, aiming to serve as a central database for Chinese language processing and in-depth linguistic research. The project focuses on building a self-sufficient ecosystem where an electronic lexicon supports corpus tagging, and both, in turn, support a parser, with initial search utilities already yielding linguistic insights.

What Problem Does This Paper Address?

Building a robust corpus for Chinese language processing presents unique challenges. Unlike many Western languages, Chinese writing systems lack explicit word boundaries, making segmentation and analysis more complex. Furthermore, the need to support both historical (classical) and contemporary (modern) Chinese requires distinct approaches to data acquisition and organization. This paper addresses the foundational work of constructing such a corpus, establishing a framework for its integration with lexical resources and parsing tools, and developing initial functionalities for linguistic research.

Methods

The project adopts a modular, mutually supportive approach. The core components are:

  • An On-line Lexicon for NLP: This lexicon is being developed with increasing grammatical detail, aiming for a comprehensive resource to automatically tag the corpus.
  • A Corpus: This serves as the primary data source, designed to be balanced and comprehensive for both modern and classical Chinese.
  • A Parser: This utilizes an information-based case grammar formalism, leveraging the lexicon and corpus for its development.

For corpus access, the paper highlights the development of a KWIC (Key-Word-In-Context) search utility that operates on Chinese characters. This utility allows for flexible context specification and includes specialized commands for linguistic analysis, such as identifying reduplication and co-occurrence patterns. The project also emphasizes the development of concordance programs and tools for frequency counts and statistical analysis.

Datasets

The paper details the acquisition of data for both modern and classical Chinese corpora:

  • Modern Chinese Corpus:
    • Approximately 10 million characters from three months of the Liberal Times newspaper.
  • 10 million characters from the China Times group (agreement secured for daily on-line text).
  • About one million words of previously input data from the CKIP project (magazine articles and dictionary explanations).
  • 30 thousand words of transcribed spoken text (to be input in 1992).
  • Classical Corpus:
    • Approximately 1.5 million characters of Chinese written prose from before 0 B.C. (available from a previous project).
  • The remaining 1.5 million characters are being keyed in and will be available by mid-1992.

Key Insights and Ideas

  • The Interdependence of Lexicon, Corpus, and Parser: The project's strength lies in its vision of these three components working in synergy, creating a self-sufficient environment for Chinese NLP. The corpus feeds the lexicon, the lexicon tags the corpus, and both support the parser.
  • Strategic Separation of Modern and Classical Chinese: This decision allows for tailored data acquisition and analysis, while also enabling interesting diachronic comparisons.
  • Character-Based Search as a Foundation: Given the lack of word boundaries, initial search utilities are character-based, providing a flexible starting point for linguistic exploration.
  • Customized Search Commands for Linguistic Discovery: The development of specific commands (e.g., for reduplication, co-occurrence, and disassociation) demonstrates a practical approach to enabling linguists to extract meaningful patterns from the data.
  • Human Expertise Driving Tool Development: The paper highlights the productive interaction between linguists and the corpus, where human expertise guides the design of search utilities and the corpus provides empirical evidence for linguistic generalizations.

Key Results

While the paper focuses on the construction and initial functionalities, it reports on:

  • The establishment of a mutual-supporting environment between lexical computing and computational linguistics research.
  • The development of a basic KWIC search utility and specialized commands that have already facilitated several linguistic studies.
  • Examples of linguistic studies conducted using the corpus, including work on quantifier-measure compounds, morphological rules, and serial verb constructions.
  • The development of concordance programs and tools for frequency counts, with some modules already tested on untagged classical Chinese data.

Limitations and Future Work

Limitations:

  • The current search functions are primarily character-based, as automatic tagging and word segmentation modules are still under development.
  • The ordering of data in search results is based on traditional stroke counts, which is acknowledged as not ideal.
  • The classical corpus is defined by time, while the modern corpus is defined by size and domain, which sets a direction for future expansion.

Future Work:

  • Incorporation of an automatic tagger and extraction of grammatical information from tagged corpora.
  • Development of a theoretically well-structured attribute system for tagging, potentially following TEI guidelines.
  • Refinement of the word segmentation program to serve as a more robust category-tagger.
  • Identification of more search functions and refinement of existing utilities based on ongoing linguistic research.
  • Direct extraction of dictionaries and grammars from the corpus, projected to be feasible within five years.
  • Porting of developed modules to IBM PC 486 machines running Unix for collaborators.

Summary updated

BibTeX
@inproceedings{huang1992chinese,
  author = {Chu-Ren Huang and Keh-jiann Chen},
  title = {{A Chinese Corpus for Linguistic Research}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1992},
}
Figure