A Fine-grained Chinese Software Privacy Policy Dataset for Sequence Labeling and Regulation Compliant Identification

Kaifa Zhao, Le Yu, Shiyao Zhou, Jing Li, Xiapu Luo, Aemon Yat Fei Chiu, Yutong Liu

Introduction

Privacy policies are essential legal documents that outline how software applications handle user data. However, these documents are frequently criticized for being overly long, laden with legal jargon, and intentionally vague. While Natural Language Processing (NLP) has been employed to help users and regulators parse these documents, existing research faces two significant hurdles. First, there is a distinct language bias; most high-quality datasets are in English, leaving the vast Chinese software ecosystem underserved. Second, existing datasets often suffer from coarse granularity, labeling entire sentences rather than specific entities. To verify compliance with regulations like the GDPR or China’s Personal Information Security Specification (PISS), a more precise approach is required to identify exactly who is collecting what data, for what purpose, and with whom it is shared.

This paper introduces CA4P-483, the first large-scale, fine-grained Chinese dataset for software privacy policies. By shifting from sentence-level classification to character-level sequence labeling, this dataset enables models to extract the specific components necessary for regulatory auditing.

The CA4P-483 Dataset

The core contribution of this work is the CA4P-483 dataset, which consists of 483 privacy policies from Chinese Android applications. The dataset includes over 11,000 sentences and 52,000 fine-grained annotations.

Unlike the widely used English dataset OPP-115, which operates at the sentence level, CA4P-483 utilizes character-level annotations. This high resolution allows for the identification of seven specific categories critical to legal compliance:

  • Data Controller: The entity responsible for data collection.
  • Data Entity: The specific type of data being collected (e.g., location, contacts).
  • Collecting Action: The act of gathering data.
  • Sharing Action: The act of transferring data to others.
  • Condition: The circumstances under which data is processed.
  • Purpose: The reason for the data collection or processing.
  • Data Receiver: The third parties receiving the data.

Methodology

The researchers framed the analysis of privacy policies as a Chinese Software Privacy Policy (CSP) task, treated as a specialized form of Named Entity Recognition (NER). They evaluated several sequence labeling architectures to establish benchmarks for the dataset:

  1. Probabilistic Models: Hidden Markov Models (HMM) and Conditional Random Fields (CRF).
  2. Neural Networks: Bi-directional Long Short-Term Memory (BiLSTM) and BiLSTM-CRF.
  3. Pre-trained Models: BERT-BiLSTM-CRF.
  4. Lattice-based Models: Lattice-LSTM, which incorporates a lexicon to handle the lack of word boundaries in Chinese, allowing the model to leverage dictionary knowledge during character-level processing.

Key Results and Insights

The experimental results provided several surprising insights into the nature of legal NLP in the Chinese context:

  • Top Performance: The BiLSTM-CRF model emerged as the top performer, achieving an overall F1-score of 86.84%.
  • The BERT Domain Shift: Surprisingly, the BERT-BiLSTM-CRF model performed significantly worse than the standard BiLSTM-CRF. The authors attribute this to a massive domain shift; general-purpose BERT, trained on standard Chinese text, struggles with the specialized "legalese" and technical terminology found in privacy policies.
  • Label Complexity: Models were highly accurate at identifying "Data Receivers" (92.19% F1) but struggled with "Collecting Actions" (54.47% F1). The latter often shared overlapping terminology with "Sharing Actions," leading to model confusion. Furthermore, identifying "Purpose" and "Condition" proved difficult because these entities are often expressed as long, complex clauses rather than discrete words.
  • Lattice Benefits: The Lattice-LSTM model demonstrated that incorporating a lexicon helps the model navigate word boundaries, though it occasionally struggled with the structural complexity of legal clauses.

Regulatory Compliance and Real-world Gaps

To demonstrate the practical utility of CA4P-483, the researchers used their trained models to analyze existing apps for regulatory gaps. They identified several instances of non-compliance. For example, a weather application was found to claim data collection in its policy while completely omitting the "Purpose" or "Condition" for that collection—a direct violation of Chinese transparency regulations. This highlights the potential for automated systems to assist regulators in large-scale auditing.

Limitations and Future Work

Despite the high performance of the BiLSTM-CRF model, challenges remain. One primary difficulty is the presence of overlapping components, where a single phrase may serve multiple roles (e.g., "login" acting as both a Data Entity and a Purpose). Additionally, the current models struggle with non-linear text formats, such as bulleted lists and enumeration, which are common in legal documents.

The authors suggest that the next frontier is App Behavior Consistency Identification. By comparing the results of NLP-based policy analysis with static or dynamic code analysis of the app itself, researchers can automatically detect "privacy policy inflation"—where an app's actual data practices differ from its legal claims.

Summary updated

BibTeX
@inproceedings{zhao2022fine,
  author = {Kaifa Zhao and Le Yu and Shiyao Zhou and Jing Li and Xiapu Luo and Aemon Yat Fei Chiu and Yutong Liu},
  title = {{A Fine-grained Chinese Software Privacy Policy Dataset for Sequence Labeling and Regulation Compliant Identification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.700},
}
Figure