APPCorp: a corpus for Android privacy policy document structure analysis

Shuang Liu, Fan Zhang, Baiyang Zhao, Renjie Guo, Tao Chen, Meishan Zhang

TL;DR

APPCorp introduces a new, manually annotated corpus of 167 Android privacy policies designed to help machines understand high-level document structure. By mapping paragraphs to 11 GDPR-aligned categories using a hierarchical BERT-based model, the researchers provide a benchmark for automating the outlining and restructuring of these notoriously dense legal documents.

What Problem Does This Paper Address?

We’ve all seen the "I agree" checkbox. Most of us click it without reading the privacy policy, and for good reason: reading every policy for the sites and apps we visit would take the average user roughly 244 hours per year. These documents are long, vague, and often poorly structured.

While previous research has looked at fine-grained details (like specific data types collected), there has been a lack of focus on topical structure—identifying what a whole paragraph is actually about. Furthermore, the legal landscape shifted significantly with the General Data Protection Regulation (GDPR). Older datasets don't account for new mandatory disclosures, such as international data transfer rights. APPCorp aims to bridge this gap by providing a modern, structure-focused dataset that reflects current legal requirements.

Methods

The authors frame the problem as a paragraph-level multi-class classification task. To solve it, they propose a hierarchical model architecture consisting of three parts:

  1. Sentence Representation: They compared a standard Bidirectional Long Short-Term Memory (BiLSTM) network against BERT (Bidirectional Encoder Representations from Transformers). BERT, being pre-trained on massive amounts of text, is better at capturing the nuanced semantic meaning of legal jargon.
  2. Paragraph Representation: Once sentences are encoded, the model must aggregate them. The authors tested a simple attention mechanism (P-Attn) against a second BiLSTM layer (P-BiLSTM). The P-BiLSTM approach treats the paragraph as a sequence of sentences, capturing how the flow of information contributes to the overall topic.
  3. Decoding: A standard Softmax layer produces the final label.

The most successful architecture, BERT + P-BiLSTM, is illustrated below:

Figure 4: The BERT + P-BiLSTM Model Structure

Datasets

The researchers curated a new dataset called APPCorp:

  • Source: 167 high-quality privacy policies crawled from Google Play across 22 different app categories.
  • Scale: Over 447,000 words, 15,614 sentences, and 5,167 annotated paragraphs.
  • Labels: 11 categories based on GDPR articles and expert knowledge, including First Party Collection and Use, International Data Transfer, and User Right and Control.
  • Quality: Each document was labeled by three annotators, achieving a Fleiss’ Kappa score between 0.68 and 0.80, indicating substantial agreement.

Key Insights and Ideas

  • Legislation Drives Content: The authors found that GDPR-specific labels like "International Data Transfer" appeared in nearly half of the policies. This confirms that privacy policies are evolving documents, and NLP tools must evolve with them.
  • Sequence Matters in Paragraphs: The fact that P-BiLSTM outperformed simple attention suggests that the order of sentences within a paragraph is a strong signal for its topic. A paragraph isn't just a "bag of sentences."
  • The "Short Paragraph" Hurdle: BERT-based models actually struggled slightly with very short paragraphs (under 3 sentences) compared to keyword-heavy BiLSTM models. The authors suggest this is because the BERT models weren't adequately trained on these brief snippets, whereas the BiLSTM could latch onto specific keywords like "protect" or "encrypt."
  • Contextual Overlap: Many misclassifications occurred because paragraphs naturally overlap. For example, a paragraph about "First Party Collection" might briefly mention "Third Party Sharing," confusing the model.

Key Results

The benchmark results clearly show the superiority of combining pre-trained transformers with recurrent layers for structure analysis.

  • Best Model: The BERT + P-BiLSTM model achieved a Micro-F1 of 80.56% and a Macro-F1 of 80.63%.
  • BERT vs. BiLSTM: Replacing the BiLSTM sentence encoder with BERT provided a significant performance boost across almost all categories.
  • Sentence Count Impact: Performance generally improved as the number of sentences in a paragraph increased, peaking at paragraphs with 4 sentences.
Figure 5: F1-measure against Sentence Count for BERT and H-BERT

Detailed precision, recall, and F1 scores for all four tested models can be found in the table below:

Table 3: The precision/recall/F score for 4 classification models

Limitations and Future Work

While APPCorp is a significant step forward, the authors note a few areas for improvement:

  • Data Volume: 167 documents is a solid start for manual annotation, but larger datasets would likely help BERT-based models generalize even better, especially for rare labels.
  • Multi-label Paragraphs: Currently, the task assumes one topic per paragraph. In reality, legal text is messy, and some paragraphs may require multiple labels.
  • Applications: The ultimate goal is to use these models to build tools that automatically "restructure" or "outline" policies for users, making the most important information (like how your data is transferred) impossible to miss.

Summary updated

Abstract

With the increasing popularity of mobile devices and the wide adoption of mobile Apps, an increasing concern of privacy issues is raised. Privacy policy is identified as a proper medium to indicate the legal terms, such as the general data protection regulation (GDPR), and to bind legal agreement between service providers and users. However, privacy policies are usually long and vague for end users to read and understand. It is thus important to be able to automatically analyze the document structures of privacy policies to assist user understanding. In this work we create a manually labelled corpus containing 231 privacy policies (of more than 566,000 words and 7,748 annotated paragraphs). We benchmark our data corpus with 3 document classification models and achieve more than 82% on F1-score.
BibTeX
@article{liu2023appcorp,
  author = {Shuang Liu and Fan Zhang and Baiyang Zhao and Renjie Guo and Tao Chen and Meishan Zhang},
  title = {{APPCorp: a corpus for Android privacy policy document structure analysis}},
  journal = {Frontiers of Computer Science},
  year = {2023},
  doi = {10.1007/s11704-022-1627-2},
}
Figure