Federated Chinese Word Segmentation with Global Character Associations

Yuanhe Tian, Guimin Chen, Han Qin, Yan SongACL 2021

TL;DR

This paper tackles the challenge of Chinese Word Segmentation (CWS) in real-world scenarios where data is siloed due to privacy or legal concerns. They propose GCA-FL, a federated learning approach enhanced with a Global Character Association (GCA) mechanism to effectively leverage distributed data for improved CWS performance.

What Problem Does This Paper Address?

Chinese Word Segmentation (CWS) is a crucial preprocessing step for many Natural Language Processing (NLP) tasks involving Chinese text. A persistent challenge in CWS is handling out-of-vocabulary (OOV) words, especially when the data comes from diverse sources. In real-world applications, such as clinical or financial reports, data from different institutions is often isolated due to privacy and legal restrictions, making it inaccessible for traditional centralized training. While Federated Learning (FL) offers a solution by enabling model training on decentralized data without direct access, standard FL struggles with domain diversity. CWS, in particular, is sensitive to data source variations in vocabulary and expression patterns. This paper addresses the gap of applying FL to CWS in a way that accounts for this domain heterogeneity and improves localized prediction.

Methods

The authors propose GCA-FL, a federated learning framework for CWS augmented with a Global Character Association (GCA) mechanism.

  • Federated Learning (FL): The core FL setup involves multiple client nodes (each holding isolated data) and a central server. Nodes encrypt their data, train locally, and send model updates (encrypted information like encrypted data, word segmentation tags, and loss) to the server. The server aggregates these updates to improve a centralized model. This ensures data privacy as raw data never leaves the local nodes.
  • Global Character Association (GCA): This mechanism is designed to capture contextual information from n-grams within the input text and use it to guide the centralized model for better localized predictions. GCA consists of:
  • A lexicon of global character n-grams.
  • An n-gram embedding matrix to map n-grams to embeddings.
  • A position embedding matrix to map the position of a character within an n-gram (e.g., beginning, middle, end) to an embedding. GCA extracts relevant n-grams and their positional information for each character in an input sequence. It then computes weights for these associations and combines them with the character's hidden representation from the backbone encoder (like BERT or ZEN 2.0). This enriched representation is then fed into a CRF decoder for word segmentation. This approach aims to help the model understand the context and nuances of different data sources, even when they are isolated.

Datasets

The experiments were conducted on a simulated environment using data from the CTB7 dataset. The data was split into five genres (broadcast conversation (BC), broadcast news (BN), magazine (MZ), newswire (NW), and weblog (WEB)), with each genre assigned to one of the five isolated nodes. This setup mimics real-world scenarios where data from different domains is siloed.

Key Insights and Ideas

  • FL is beneficial for CWS with isolated data: The study confirms that federated learning can effectively leverage data from multiple isolated sources to improve CWS performance compared to training on a single node.
  • Domain diversity is a critical challenge for FL in CWS: Standard FL struggles when data from different nodes has distinct characteristics, leading to suboptimal localized predictions.
  • GCA effectively bridges the domain gap: The proposed GCA mechanism significantly enhances FL by capturing contextual n-gram information and using it to adapt the centralized model to localized prediction needs, outperforming standard FL baselines.
  • GCA improves OOV handling: The GCA mechanism demonstrably improves the recall of out-of-vocabulary words, a key challenge in CWS, by better leveraging information from other nodes.
  • Union model as an upper bound: The "Union" model, trained on all data combined without isolation, serves as a strong baseline and shows that GCA-FL can achieve competitive results even against this ideal scenario.

Key Results

  • GCA-FL consistently outperforms the standard FL baseline across all five nodes (genres) for both BERT and ZEN 2.0 encoders.
  • The proposed GCA-FL approach achieves competitive results compared to the "Union" reference model (which has access to all data) in most cases.
  • The paper highlights that GCA-FL improves the identification of out-of-vocabulary (OOV) words, with the most significant improvement observed on the newswire (NW) node.
  • For example, using BERT, GCA-FL achieved F1 scores of 97.76, 97.40, 96.74, 98.43, and 95.29 on BC, BN, MZ, NW, and WB respectively, outperforming the FL baseline on each node. Similar trends were observed with the ZEN 2.0 encoder.

Limitations and Future Work

  • Simulated Environment: The experiments were conducted in a simulated FL environment. While this setup effectively mimics data isolation, real-world FL deployments might introduce additional complexities.
  • Focus on CWS: The proposed GCA-FL framework is specifically designed for CWS. Future work could explore its applicability to other NLP tasks that also suffer from data isolation and domain diversity.
  • Potential for further optimization: While GCA-FL shows strong performance, there's always room for further optimization of the GCA mechanism or integration with other advanced FL techniques to enhance efficiency and robustness.

Summary updated

BibTeX
@inproceedings{tian2021federated,
  author = {Yuanhe Tian and Guimin Chen and Han Qin and Yan Song},
  title = {{Federated Chinese Word Segmentation with Global Character Associations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure