TL;DR
APPCorp introduces a new, manually annotated corpus of 167 Android privacy policies designed to help machines understand high-level document structure. By mapping paragraphs to 11 GDPR-aligned categories using a hierarchical BERT-based model, the researchers provide a benchmark for automating the outlining and restructuring of these notoriously dense legal documents.
What Problem Does This Paper Address?
We’ve all seen the "I agree" checkbox. Most of us click it without reading the privacy policy, and for good reason: reading every policy for the sites and apps we visit would take the average user roughly 244 hours per year. These documents are long, vague, and often poorly structured.
While previous research has looked at fine-grained details (like specific data types collected), there has been a lack of focus on topical structure—identifying what a whole paragraph is actually about. Furthermore, the legal landscape shifted significantly with the General Data Protection Regulation (GDPR). Older datasets don't account for new mandatory disclosures, such as international data transfer rights. APPCorp aims to bridge this gap by providing a modern, structure-focused dataset that reflects current legal requirements.
Methods
The authors frame the problem as a paragraph-level multi-class classification task. To solve it, they propose a hierarchical model architecture consisting of three parts:
- Sentence Representation: They compared a standard Bidirectional Long Short-Term Memory (BiLSTM) network against BERT (Bidirectional Encoder Representations from Transformers). BERT, being pre-trained on massive amounts of text, is better at capturing the nuanced semantic meaning of legal jargon.
- Paragraph Representation: Once sentences are encoded, the model must aggregate them. The authors tested a simple attention mechanism (P-Attn) against a second BiLSTM layer (P-BiLSTM). The P-BiLSTM approach treats the paragraph as a sequence of sentences, capturing how the flow of information contributes to the overall topic.
- Decoding: A standard Softmax layer produces the final label.
The most successful architecture, BERT + P-BiLSTM, is illustrated below:
Datasets
The researchers curated a new dataset called APPCorp:
- Source: 167 high-quality privacy policies crawled from Google Play across 22 different app categories.
- Scale: Over 447,000 words, 15,614 sentences, and 5,167 annotated paragraphs.
- Labels: 11 categories based on GDPR articles and expert knowledge, including First Party Collection and Use, International Data Transfer, and User Right and Control.
- Quality: Each document was labeled by three annotators, achieving a Fleiss’ Kappa score between 0.68 and 0.80, indicating substantial agreement.
Key Insights and Ideas
- Legislation Drives Content: The authors found that GDPR-specific labels like "International Data Transfer" appeared in nearly half of the policies. This confirms that privacy policies are evolving documents, and NLP tools must evolve with them.
- Sequence Matters in Paragraphs: The fact that P-BiLSTM outperformed simple attention suggests that the order of sentences within a paragraph is a strong signal for its topic. A paragraph isn't just a "bag of sentences."
- The "Short Paragraph" Hurdle: BERT-based models actually struggled slightly with very short paragraphs (under 3 sentences) compared to keyword-heavy BiLSTM models. The authors suggest this is because the BERT models weren't adequately trained on these brief snippets, whereas the BiLSTM could latch onto specific keywords like "protect" or "encrypt."
- Contextual Overlap: Many misclassifications occurred because paragraphs naturally overlap. For example, a paragraph about "First Party Collection" might briefly mention "Third Party Sharing," confusing the model.
Key Results
The benchmark results clearly show the superiority of combining pre-trained transformers with recurrent layers for structure analysis.
- Best Model: The BERT + P-BiLSTM model achieved a Micro-F1 of 80.56% and a Macro-F1 of 80.63%.
- BERT vs. BiLSTM: Replacing the BiLSTM sentence encoder with BERT provided a significant performance boost across almost all categories.
- Sentence Count Impact: Performance generally improved as the number of sentences in a paragraph increased, peaking at paragraphs with 4 sentences.
Detailed precision, recall, and F1 scores for all four tested models can be found in the table below:
Limitations and Future Work
While APPCorp is a significant step forward, the authors note a few areas for improvement:
- Data Volume: 167 documents is a solid start for manual annotation, but larger datasets would likely help BERT-based models generalize even better, especially for rare labels.
- Multi-label Paragraphs: Currently, the task assumes one topic per paragraph. In reality, legal text is messy, and some paragraphs may require multiple labels.
- Applications: The ultimate goal is to use these models to build tools that automatically "restructure" or "outline" policies for users, making the most important information (like how your data is transferred) impossible to miss.
Summary updated