TL;DR
This paper introduces two large-scale Chinese linguistic datasets: a segmented and POS-tagged web corpus (PText) and a Chinese POS n-gram corpus (PNgram), both derived from the ClueWeb09 dataset. They also provide user-friendly toolkits to access and query these resources, aiming to facilitate research in Chinese Natural Language Processing (NLP).
What Problem Does This Paper Address?
Processing web-scale text for natural language research is computationally intensive and requires significant linguistic resources. While large text corpora exist, readily accessible, linguistically annotated datasets for Chinese at a web scale are scarce. This paper addresses this gap by creating and releasing PText and PNgram, along with tools to make them practical for NLP applications, thereby lowering the barrier to entry for researchers working with Chinese web data.
Methods
The authors developed two Chinese datasets from the ClueWeb09 collection.
- PText (Chinese Segmented and POS-Tagged Dataset):
- Preprocessing: Involved encoding detection and language identification to filter out non-Chinese pages.
- Segmentation: Chinese web pages were translated to simplified Chinese and then segmented into sentences. Sentence boundaries were identified using punctuation and newlines.
- POS Tagging: The Stanford segmenter and tagger were used, adopting the LDC Chinese Treebank POS tag set.
- PNgram (Chinese POS N-gram Corpus):
- Vocabulary Creation: Unigrams (single words) were identified with a minimum occurrence count of 200.
- N-gram Extraction: Word n-grams (up to 5-grams) were extracted with a minimum occurrence count of 40.
- POS Sequence Association: POS sequences were then extracted for each word n-gram.
The datasets are stored in Java serialization format for easy manipulation, and conversion tools to plain text are provided.
Datasets
- ClueWeb09: The source dataset, containing approximately 1 billion web pages in ten languages. The Chinese portion comprised over 173 million pages after filtering.
- PText: A newly introduced dataset containing 9,598,430,559 POS-tagged sentences across 172,298,866 documents. It preserves original metadata like title, TREC ID, and URI.
- PNgram: A newly introduced corpus derived from PText, containing POS n-grams. Statistics show a significant drop in unique n-grams as n increases, with only 2.1% of unique words appearing more than 200 times.
Key Insights and Ideas
- Web-scale linguistic resources are crucial: The paper highlights the need for large, annotated datasets to advance NLP research, especially for languages like Chinese where such resources are less common.
- Practical toolkits are as important as data: The authors emphasize that providing user-friendly tools for accessing and querying these large datasets is essential for their adoption and utility.
- Chinese NLP faces unique challenges: The paper touches upon the complexities of Chinese text processing, including character encoding, segmentation, and language identification, which require careful handling at scale.
- POS information is valuable for diverse NLP tasks: The paper suggests applications ranging from knowledge mining and sentiment analysis to basic NLP tasks and language learning, underscoring the broad utility of POS-tagged data.
Key Results
- Dataset Size: PText contains over 9.5 billion POS-tagged sentences, and PNgram is derived from this extensive resource.
- POS Tagging Accuracy: The Stanford tagger achieved 94.13% accuracy on a combination of simplified Chinese and Hong Kong texts, and 78.92% on unknown words.
- N-gram Statistics: Table 1 shows that while there are over 107 million unique words in PText, only a small fraction (2.1%) meet the minimum frequency threshold for unigrams. The number of unique n-grams decreases significantly as n increases. Table 2 shows that the average number of POS patterns per word n-gram is relatively small, ranging from 1.7 to 3.5.
Limitations and Future Work
- Limitations: The paper does not explicitly detail limitations of the datasets themselves, but the inherent challenges of web-scale data processing (e.g., noise, ambiguity) are implied. The POS tagging accuracy on unknown words (78.92%) suggests room for improvement.
- Future Work: The authors plan to study the polarity of Chinese discourse markers using these corpora. They will extract sentences containing discourse markers, measure sentiment polarities of connected arguments, and investigate the relationship between discourse parsing and sentiment analysis.
Summary updated