Abstract
This paper formulates a new task of extracting privacy parameters from a privacy policy, through the lens of Contextual Integrity, an established social theory framework for reasoning about privacy norms. Privacy policies, written by lawyers, are lengthy and often comprise incomplete and vague statements. In this paper, we show that traditional NLP tasks, including the recently proposed Question-Answering based solutions, are insufficient to address the privacy parameter extraction problem and provide poor precision and recall. We describe 4 different types of conventional methods that can be partially adapted to address the parameter extraction task with varying degrees of success: Hidden Markov Models, BERT fine-tuned models, Dependency Type Parsing (DP) and Semantic Role Labeling (SRL). Based on a detailed evaluation across 36 real-world privacy policies of major enterprises, we demonstrate that a solution combining syntactic DP coupled with type-specific SRL tasks provides the highest accuracy for retrieving contextual privacy parameters from privacy statements. We also observe that incorporating domain-specific knowledge is critical to achieving high precision and recall, thus inspiring new NLP research to address this important problem in the privacy domain.
Introduction
Privacy policies are notoriously dense, vague, and written in complex legalese, making it nearly impossible for average users to understand how their data is handled. While previous NLP research has successfully categorized paragraphs (e.g., flagging a section as "data retention"), it struggles to parse the exact, sentence-level mechanics of data flows.
To bridge this gap, this research leverages Contextual Integrity (CI), a sociological framework that defines privacy as the appropriate flow of information based on five essential parameters:
- Sender: Who is sending the data?
- Receiver: Who is getting it?
- Subject: Who is the data about?
- Attribute: What specific information is being transferred?
- Transmission Principle (TP): Under what conditions does this transfer happen?
Privacy statements often omit some of these parameters or bury them in complex syntax. This paper addresses the challenge of automatically extracting these five CI parameters from convoluted text to enable automated privacy auditing.
Methodology
The researchers evaluated several standard NLP approaches before developing a specialized hybrid method. The workflow involves processing raw privacy policies through various NLP tasks to generate CI annotations.
Evaluated Models
- Baselines (HMM, BERT, QA): Extraction was framed as a Question Answering task, a POS-tagging task using a Trigram Hidden Markov Model (HMM), and a sequence-to-sequence task using a fine-tuned BERT model.
- Dependency Parsing (DP): Syntactic roles (like direct objects or nominal subjects) were mapped to CI parameters using the Spacy parser.
- Semantic Role Labeling (SRL): Using an AllenNLP BiLSTM model, the team extracted verb-argument structures. Semantic roles (e.g.,
ARG0as agent,ARG1as item) were mapped to CI parameters based on whether the verb implied "sending" or "receiving." - Improved SRL (CI-SRL): This novel contribution addresses the high false-positive rate of standard SRL. By injecting domain knowledge—filtering for verbs specifically related to information exchange (e.g., "collect", "share")—and implementing a redundancy-resolution algorithm for nested actions, the system significantly improved precision.
Dataset
The study utilized the OPP-115 Corpus, focusing on 36 real-world privacy policies. From these, 2,268 individual sentences discussing information exchanges were isolated. A domain expert manually annotated 778 valid statements containing 3,245 ground-truth CI parameters to serve as the evaluation benchmark.
Key Results
The evaluation demonstrated that general-purpose models struggle with the specific nuances of privacy legalese:
- QA Models: Failed significantly, with F1 scores between 0.03 and 0.18.
- HMM and BERT: Struggled particularly with the "Sender" parameter (F1 of 0.09 and 0.13, respectively).
- Dependency Parsing (DP): Achieved moderate results (F1: 0.40–0.68) but suffered from low precision.
- Standard SRL: Showed improvement, balancing precision and recall (F1: 0.76–0.83).
- CI-SRL: The custom heuristic approach achieved the best overall performance, with F1 scores ranging from 0.82 to 0.87 across all five CI parameters.
Discussion and Future Work
A major insight from this work is that syntax alone is insufficient. Dependency parsing yields high recall but poor precision because grammatical roles do not consistently map to semantic privacy roles. Furthermore, domain knowledge is the critical missing link; filtering for data-exchange verbs drastically reduced false positives.
However, limitations remain. The heuristic algorithm struggles with semantically complex run-on sentences or poorly punctuated lists, which can cause the parser to generate a cascade of false positives.
Future work suggests using these tools to detect overly complex statements for human rewriting. Ultimately, the goal is to translate privacy policies into formal logic, enabling automated systems to audit corporate data flows for compliance.
Summary updated