A Step Towards Usable Privacy Policy: Automatic Alignment of Privacy Statements

Fei Liu, Rohan Ramanath, Norman M. Sadeh, Noah A. SmithCOLING 2014 · OpenReview

TL;DR

This paper tackles the challenge of making privacy policies understandable by automatically aligning segments of different policies that address the same privacy issue. They introduce a new annotated dataset and evaluate clustering and Hidden Markov Model (HMM) based methods, finding that these automated approaches can close about half the gap in agreement with human annotators, with further gains requiring more sophisticated text representations.

What Problem Does This Paper Address?

Privacy policies are notoriously long, complex, and difficult for the average user to understand. This leads to users often agreeing to terms without fully grasping the implications for their personal information. While efforts like P3P and "nutrition labels" have aimed to improve clarity, they haven't fully solved the problem. This paper focuses on a fundamental step towards making these policies more usable: automatically identifying and grouping similar privacy concerns across different policy documents. This is crucial for building tools that can summarize policies or help users compare them, ultimately empowering users to make more informed decisions about their data.

Methods

The authors explore two main approaches for aligning segments of privacy policies:

  • Clustering: They use the CLUTO algorithm, a hierarchical clustering method that iteratively bisects clusters to optimize a criterion. This is a standard approach for grouping similar items.
  • Hidden Markov Models (HMMs): They employ HMMs, a probabilistic model that assumes a sequence of hidden states (representing privacy issues) generates observed text segments (sections or paragraphs). They use two estimation methods:
  • Expectation-Maximization (EM): A standard iterative algorithm for estimating parameters of probabilistic models.
  • Variational Bayesian (VB) Inference: A method for approximating Bayesian inference, which can be more robust on smaller datasets.

A key aspect of their HMM approach is the use of multiple emissions per hidden state, allowing each state to represent a broader range of words associated with a privacy issue, and the Viterbi algorithm for decoding the most likely sequence of hidden states given an observation sequence.

Datasets

The paper utilizes a newly created dataset of 1,010 unique privacy policy documents collected from top websites. This dataset was used to generate a large number of pairwise annotations of section and paragraph pairs. These annotations were collected from crowdworkers on Amazon Mechanical Turk, who were asked to determine if two text segments discussed the same privacy issue. This dataset is an extension and refinement of previous work by Ramanath et al. (2014), with a focus on paragraph-level analysis and improved annotation quality.

Key Insights and Ideas

  • The "five-point gap": The authors identify a significant gap between the agreement of their automated methods and the consensus of human annotators. They suggest this gap is roughly split between the limitations of using simple "bag of words" representations and the inherent "greediness" of their alignment algorithms.
  • Paragraph vs. Section Alignment: While intuitively one might expect paragraph-level alignment to be easier due to their smaller scope, the results showed similar performance for both section and paragraph alignment, with HMMs slightly outperforming clustering at the paragraph level for certain numbers of hidden states.
  • Cosine Similarity as a Strong Predictor: The paper confirms that cosine similarity of TF-IDF vectors between text segments is a strong predictor of human agreement, suggesting that word-level similarity is a good starting point for this task.
  • Boilerplate Text Helps: The authors note that "boilerplate" text, common across many policies (like in Example 1 of Table 4), can simplify alignment and potentially make automated methods more effective.

Key Results

  • The automated alignment methods (clustering and HMMs) achieve agreement on par with the lower half of crowdworkers.
  • At the section level, clustering performed slightly better in terms of F-score for larger numbers of hidden states (K), while HMMs and clustering were close at K=10. CLUTO's best performance was 85% F-score at K=14.
  • At the paragraph level, HMMs outperformed clustering in the K∈ [5, 15) range, with a peak F-score of 87%.
  • The authors estimate that using more sophisticated text representations could close about half of the gap between their methods and median crowdworkers.

Limitations and Future Work

  • Bag of Words Representation: The paper acknowledges that the reliance on bag-of-words representations limits the sophistication of their similarity measures. Future work could explore more advanced techniques like word embeddings or topic models.
  • Greediness of Algorithms: The "greedy" nature of both clustering and HMM decoding means they might not find the globally optimal alignment. Exploring non-greedy approaches or more global optimization strategies could be beneficial.
  • Annotation Granularity: The authors suggest that for cases where annotator agreement is low (like Example 2 in Table 4), more careful training of annotators or a more fine-grained set of predefined concepts might be necessary.
  • Future Directions: The authors suggest exploring Bayesian topic models for discovering document sections, rather than relying on pre-defined section/paragraph boundaries, and investigating more sophisticated similarity measures beyond TF-IDF.

Summary updated

BibTeX
@inproceedings{liu2014step,
  author = {Fei Liu and Rohan Ramanath and Norman M. Sadeh and Noah A. Smith},
  title = {{A Step Towards Usable Privacy Policy: Automatic Alignment of Privacy Statements}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2014},
  url = {https://openreview.net/forum?id=NuwYaCIDPk},
}
Figure