APPCorp: a corpus for Android privacy policy document structure analysis

Shuang Liu, Fan Zhang, Baiyang Zhao, Renjie Guo, Tao Chen, Meishan Zhang

TL;DR

This paper introduces APPCorp, a novel, manually annotated corpus of Android privacy policies, designed to facilitate the automatic analysis of their document structure. By developing a classification scheme based on GDPR articles and other relevant topics, they enable paragraph-level topic categorization, which can help users better understand these lengthy and complex legal documents.

What Problem Does This Paper Address?

In today's digital landscape, mobile apps collect vast amounts of personal data, and privacy policies are the primary legal documents outlining how this data is handled. However, these policies are notoriously long, written in dense legal jargon, and often difficult for the average user to comprehend. This opacity leads to users agreeing to terms they don't fully understand, as highlighted by the example of the ZAO app where data ownership was unconditionally transferred. Existing research has focused on specific aspects like vague language or data types, but a comprehensive approach to understanding the structure and topics within these policies at a granular level, especially considering regulations like GDPR, has been lacking.

Methods

The core of this work is the creation of the APPCorp dataset and the development of a paragraph-level classification task. The authors devised an 11-label scheme for categorizing paragraphs, drawing inspiration from GDPR articles, existing privacy policy corpora (like OPP-115), and unsupervised topic modeling results. They then manually annotated 167 privacy policies, resulting in over 5,000 paragraphs and 15,000 sentences.

For the classification task, they explored four different models, combining various sentence and paragraph representation techniques. These included:

  • S-BiLSTM + P-BiLSTM: A hierarchical model using BiLSTM for sentence encoding and another BiLSTM for paragraph representation, with self-attention for aggregation.
  • S-BiLSTM + P-Attn: Similar to the above, but using an attention mechanism for paragraph representation.
  • BERT + P-BiLSTM: Leveraging BERT for powerful sentence embeddings and BiLSTM for paragraph representation.
  • BERT + P-Attn: Using BERT for sentence embeddings and an attention mechanism for paragraph representation.

The BERT-based models, particularly BERT + P-BiLSTM, demonstrated superior performance, indicating the effectiveness of pre-trained language models for this task.

Datasets

The paper introduces APPCorp, a newly created corpus of 167 Android privacy policies. This corpus contains over 447,000 words and 5,167 annotated paragraphs, with each paragraph labeled according to the 11-category scheme. This dataset is a significant contribution as it focuses on paragraph-level structural analysis, a granularity not extensively covered by previous corpora like OPP-115.

Key Insights and Ideas

  • The Importance of Structure: The research strongly suggests that analyzing the structure of privacy policies, not just their content, is crucial for user comprehension. Categorizing paragraphs into topics provides a high-level overview, akin to an abstract for each section.
  • GDPR's Influence: The study highlights how GDPR has significantly impacted privacy policies, with many companies explicitly addressing data transfer and user rights, aligning with the proposed label scheme.
  • BERT's Dominance: Pre-trained language models like BERT are highly effective for capturing nuanced semantic information in privacy policies, outperforming traditional BiLSTM-based sentence encoders.
  • Paragraph Representation Matters: The choice of paragraph representation (BiLSTM vs. attention) also plays a significant role, with BiLSTM generally showing better performance, especially when combined with BERT sentence embeddings.
  • Short Paragraphs are Tricky: The analysis revealed that very short paragraphs, particularly in the "Data Security" category, posed challenges for some models, suggesting that context and length can influence classification accuracy.

Key Results

The BERT + P-BiLSTM model achieved the best performance, with Macro-F1 and Micro-F1 scores of 80.56% and 80.63%, respectively. This model leverages BERT for sentence representation and a BiLSTM for paragraph representation. The paper also reports that the BERT-based models generally outperformed the S-BiLSTM-based models, and models using BiLSTM for paragraph representation tended to perform better than those using only attention.

Limitations and Future Work

The authors acknowledge that the models can sometimes struggle with short paragraphs or when descriptions subtly blend topics (e.g., "Data Security" vs. "International Data Transfer"). They also note that performance on categories with fewer test samples can be more sensitive to small variations.

Future work could involve:

  • Expanding the corpus to include more policies and potentially other languages.
  • Investigating more sophisticated paragraph representation techniques.
  • Exploring how this paragraph-level labeling can be directly used to automatically outline or restructure privacy policies for better user experience.
  • Further analyzing the impact of specific GDPR clauses on policy structure and content.

Summary updated

Abstract

With the increasing popularity of mobile devices and the wide adoption of mobile Apps, an increasing concern of privacy issues is raised. Privacy policy is identified as a proper medium to indicate the legal terms, such as the general data protection regulation (GDPR), and to bind legal agreement between service providers and users. However, privacy policies are usually long and vague for end users to read and understand. It is thus important to be able to automatically analyze the document structures of privacy policies to assist user understanding. In this work we create a manually labelled corpus containing 231 privacy policies (of more than 566,000 words and 7,748 annotated paragraphs). We benchmark our data corpus with 3 document classification models and achieve more than 82% on F1-score.
BibTeX
@article{liu2023appcorp,
  author = {Shuang Liu and Fan Zhang and Baiyang Zhao and Renjie Guo and Tao Chen and Meishan Zhang},
  title = {{APPCorp: a corpus for Android privacy policy document structure analysis}},
  journal = {Frontiers of Computer Science},
  year = {2023},
  doi = {10.1007/s11704-022-1627-2},
}
Figure