Improving Chinese Segmentation-free Word Embedding With Unsupervised Association Measure

Yifan Zhang, Maohua Wang, Yongjian Huang, Qianrong Gu

Abstract

Recent work on segmentation-free word embedding (sembei) developed a new pipeline of word embedding for unsegmented language while avoiding segmentation as a preprocessing step. However, too many noisy n-grams existing in the embedding vocabulary that do not have strong association strength between characters would limit the quality of learned word embedding. To deal with this problem, a new version of segmentation-free word embedding model is proposed by collecting n-grams vocabulary via a novel unsupervised association measure called pointwise association with times information (PATI). Comparing with the commonly used n-gram filtering method like frequency used in sembei and pointwise mutual information (PMI), the proposed method leverages more latent information from the corpus and thus is able to collect more valid n-grams that have stronger cohesion as embedding targets in unsegmented language data, such as Chinese texts. Further experiments on Chinese SNS data show that the proposed model improves performance of word embedding in downstream tasks.

Introduction

In languages like Chinese, where words are not separated by spaces, Natural Language Processing (NLP) typically begins with a Chinese Word Segmentation (CWS) step. This process is often error-prone, particularly when dealing with informal text, technical jargon, or rapidly evolving social media slang. Errors in segmentation lead to "cascading errors" in downstream tasks like sentiment analysis or named entity recognition.

While "segmentation-free" models like sembei have been proposed to bypass this bottleneck, they often rely on raw frequency to select n-grams for embedding. This results in a vocabulary cluttered with "noisy" fragments—sequences of characters that appear frequently together but do not form meaningful semantic units. To address this, we introduce SGNS-PFNE (Skip-gram Negative Sampling with PATI Filtered N-grams Embedding), which utilizes a novel unsupervised association measure to ensure high-quality embedding vocabularies.

Methodology: The PATI Filter

The core of our approach is the PATI (Pointwise Association with Times Information) measure. Unlike simple frequency counts, PATI evaluates the "cohesion" or association strength of an n-gram. It is composed of three distinct factors:

  1. Frequency ($F$): The raw occurrence count of the n-gram in the corpus.
  2. Modified PMI ($MP$): A variation of Pointwise Mutual Information that emphasizes local association, identifying how strongly the constituent parts of an n-gram "stick" together.
  3. Association with Times Information ($AT$): A novel metric that calculates the "statistical uniqueness" of a character combination. It assesses how often a specific combination appears relative to all other possible combinations involving those characters, effectively penalizing "social butterfly" characters that appear in many meaningless fragments.

The SGNS-PFNE pipeline follows a structured workflow to transform raw text into high-quality embeddings without explicit segmentation.

The SGNS-PFNE pipeline: from candidate n-gram collection to final evaluation.

As shown in Figure 1, the process begins by collecting all possible n-grams (typically length 1–6). These candidates are then scored using the PATI measure. The top-$K$ scoring n-grams form the vocabulary, which is used to construct an n-gram lattice for the training corpus. Finally, a standard Skip-gram model is trained on this lattice.

Experimental Setup

We evaluated our model using a 100MB Chinese SNS dataset containing approximately 40 million characters. This dataset represents a significant challenge due to its informal nature and high density of neologisms. To verify the validity of the selected n-grams, we compared them against the Jieba dictionaries (both "basic" and "rich" versions).

For downstream performance, we conducted a Noun Category Prediction task using categories extracted from Wikidata. This task requires the model to classify words into semantic categories (e.g., "city," "job," "chemical compound"), serving as a proxy for the quality of the learned semantic space.

Results and Discussion

Our results indicate that PATI significantly outperforms both frequency-based (sembei) and PMI-based filtering methods in identifying valid Chinese words.

Vocabulary Quality

In the SNS dataset, PATI identified 12.6% more valid words than the frequency-based method when compared against a basic dictionary, and 16.7% more when compared against a rich dictionary. This suggests that PATI is far more effective at capturing the "long tail" of valid words that may not have extremely high raw frequencies.

Precision-Recall Performance

The superiority of the PATI filter is further evidenced by the Precision-Recall curves.

Precision-Recall curves comparing SGNS-PFNE (PATI) against SGNS-sembei (Frequency) and SGNS-PMI on the basic dictionary.

As illustrated in Figure 4, SGNS-PFNE maintains significantly higher precision across various recall levels compared to its counterparts. This indicates that the n-grams prioritized by PATI are much more likely to be legitimate words.

Downstream Task Performance

In the Noun Category Prediction task, SGNS-PFNE achieved an F1 score of 0.742, outperforming SGNS-sembei (0.705) and even a standard Skip-Gram model using a pre-trained segmenter (0.669). This confirms that a cleaner, cohesion-based vocabulary leads to more robust semantic representations.

Conclusion

By shifting the focus from raw frequency to unsupervised association strength, the PATI filter provides a powerful tool for generating Chinese word embeddings without the need for error-prone segmentation. This approach is particularly valuable for informal and domain-specific corpora where traditional segmenters fail. Future work will explore integrating PATI with transformer-based architectures like BERT and incorporating sub-character information to further refine embedding quality.

Summary updated

Abstract

Recent work on segmentation-free word embedding(sembei) developed a new pipeline of word embedding for unsegmentated language while avoiding segmentation as a preprocessing step. However, too many noisy n-grams existing in the embedding vocabulary that do not have strong association strength between characters would limit the quality of learned word embedding. To deal with this problem, a new version of segmentation-free word embedding model is proposed by collecting n-grams vocabulary via a novel unsupervised association measure called pointwise association with times information(PATI). Comparing with the commonly used n-gram filtering method like frequency used in sembei and pointwise mutual information(PMI), the proposed method leverages more latent information from the corpus and thus is able to collect more valid n-grams that have stronger cohesion as embedding targets in unsegmented language data, such as Chinese texts. Further experiments on Chinese SNS data show that the proposed model improves performance of word embedding in downstream tasks.
BibTeX
@misc{zhang2020improving,
  author = {Yifan Zhang and Maohua Wang and Yongjian Huang and Qianrong Gu},
  title = {{Improving Chinese Segmentation-free Word Embedding With Unsupervised Association Measure}},
  year = {2020},
  eprint = {2007.02342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.02342},
}
Figure