TL;DR
This paper introduces APPCorp, a novel, manually annotated corpus of Android privacy policies, designed to facilitate the automatic analysis of their document structure. By developing a classification scheme based on GDPR articles and other relevant topics, they enable paragraph-level topic categorization, which can help users better understand these lengthy and complex legal documents.
What Problem Does This Paper Address?
In today's digital landscape, mobile apps collect vast amounts of personal data, and privacy policies are the primary legal documents outlining how this data is handled. However, these policies are notoriously long, written in dense legal jargon, and often difficult for the average user to comprehend. This opacity leads to users agreeing to terms they don't fully understand, as highlighted by the example of the ZAO app where data ownership was unconditionally transferred. Existing research has focused on specific aspects like vague language or data types, but a comprehensive approach to understanding the structure and topics within these policies at a granular level, especially considering regulations like GDPR, has been lacking.
Methods
The core of this work is the creation of the APPCorp dataset and the development of a paragraph-level classification task. The authors devised an 11-label scheme for categorizing paragraphs, drawing inspiration from GDPR articles, existing privacy policy corpora (like OPP-115), and unsupervised topic modeling results. They then manually annotated 167 privacy policies, resulting in over 5,000 paragraphs and 15,000 sentences.
For the classification task, they explored four different models, combining various sentence and paragraph representation techniques. These included:
- S-BiLSTM + P-BiLSTM: A hierarchical model using BiLSTM for sentence encoding and another BiLSTM for paragraph representation, with self-attention for aggregation.
- S-BiLSTM + P-Attn: Similar to the above, but using an attention mechanism for paragraph representation.
- BERT + P-BiLSTM: Leveraging BERT for powerful sentence embeddings and BiLSTM for paragraph representation.
- BERT + P-Attn: Using BERT for sentence embeddings and an attention mechanism for paragraph representation.
The BERT-based models, particularly BERT + P-BiLSTM, demonstrated superior performance, indicating the effectiveness of pre-trained language models for this task.
Datasets
The paper introduces APPCorp, a newly created corpus of 167 Android privacy policies. This corpus contains over 447,000 words and 5,167 annotated paragraphs, with each paragraph labeled according to the 11-category scheme. This dataset is a significant contribution as it focuses on paragraph-level structural analysis, a granularity not extensively covered by previous corpora like OPP-115.
Key Insights and Ideas
- The Importance of Structure: The research strongly suggests that analyzing the structure of privacy policies, not just their content, is crucial for user comprehension. Categorizing paragraphs into topics provides a high-level overview, akin to an abstract for each section.
- GDPR's Influence: The study highlights how GDPR has significantly impacted privacy policies, with many companies explicitly addressing data transfer and user rights, aligning with the proposed label scheme.
- BERT's Dominance: Pre-trained language models like BERT are highly effective for capturing nuanced semantic information in privacy policies, outperforming traditional BiLSTM-based sentence encoders.
- Paragraph Representation Matters: The choice of paragraph representation (BiLSTM vs. attention) also plays a significant role, with BiLSTM generally showing better performance, especially when combined with BERT sentence embeddings.
- Short Paragraphs are Tricky: The analysis revealed that very short paragraphs, particularly in the "Data Security" category, posed challenges for some models, suggesting that context and length can influence classification accuracy.
Key Results
The BERT + P-BiLSTM model achieved the best performance, with Macro-F1 and Micro-F1 scores of 80.56% and 80.63%, respectively. This model leverages BERT for sentence representation and a BiLSTM for paragraph representation. The paper also reports that the BERT-based models generally outperformed the S-BiLSTM-based models, and models using BiLSTM for paragraph representation tended to perform better than those using only attention.
Limitations and Future Work
The authors acknowledge that the models can sometimes struggle with short paragraphs or when descriptions subtly blend topics (e.g., "Data Security" vs. "International Data Transfer"). They also note that performance on categories with fewer test samples can be more sensitive to small variations.
Future work could involve:
- Expanding the corpus to include more policies and potentially other languages.
- Investigating more sophisticated paragraph representation techniques.
- Exploring how this paragraph-level labeling can be directly used to automatically outline or restructure privacy policies for better user experience.
- Further analyzing the impact of specific GDPR clauses on policy structure and content.
Summary updated