Abstract
Privacy policies are essential legal documents that outline how companies handle user data, yet they are notoriously difficult to read due to intentionally ambiguous language. This paper presents the first human-annotated corpus specifically designed for vagueness detection. We evaluate several deep learning architectures, including context-aware Bi-LSTMs for word-level detection and Auxiliary-Classifier GANs (AC-GANs) for sentence-level classification. Our results demonstrate that context is crucial for identifying vagueness and that generative models can effectively augment training data in this specialized domain.
Introduction
Most internet users "agree" to privacy policies without reading them, largely because these documents are lengthy and written in an "opaque" style. Legal counsel often employs vague terms (e.g., "we may share some information") to maintain flexibility as company practices evolve. This research, conducted by Lebanoff and Liu at the University of Central Florida, aims to quantify and automatically detect this vagueness. By providing tools to flag ambiguous language, the authors hope to help users navigate these documents and enable regulators to hold companies accountable for their terms of service.
Methodology
The authors address vagueness at two distinct levels: word-level and sentence-level.
Word-Level Detection
The study compares context-agnostic models against context-aware Bi-LSTMs. While a word like "may" might seem inherently vague, its meaning depends heavily on context—for instance, "Users may post" (granting permission) is clear, whereas "We may share data" (indicating a possibility) is vague. The context-aware model utilizes word2vec embeddings to capture these semantic nuances, significantly outperforming simpler models.
Sentence-Level Classification
For sentence-level analysis, the authors employ an Auxiliary-Classifier GAN (AC-GAN). This architecture includes a generator that produces synthetic privacy policy sentences and a multi-task discriminator. The discriminator is trained to distinguish real from fake sentences while simultaneously classifying them into four vagueness categories: Clear, Somewhat Clear, Vague, and Extremely Vague.
To bridge the gap between discrete text and the differentiable requirements of GANs, the authors use the "Straight-Through Gumbel-Softmax" trick, allowing the model to backpropagate through the discrete sampling process of the generator.
Dataset and Annotation
A new benchmark corpus was created, consisting of 133,000 words and 4,500 sentences from 100 website privacy policies. Human annotators on Amazon Mechanical Turk highlighted vague terms and scored sentences on a 1-5 scale.
The dataset reveals that while vagueness is subjective, there is significant human agreement. Nearly 47% of sentences received consistent scores from three or more annotators, providing a solid ground truth for the task.
Key Insights and Results
- Context is King: The context-aware Bi-LSTM achieved an F1-score of 60.08%, whereas the context-agnostic model suffered from extremely low precision (11%), confirming that vagueness is a contextual property.
- The Noun Blind Spot: Models struggled most with nouns like "information" or "services." While humans often flag these as vague when undefined, models tended to focus on adjectives and adverbs.
- GANs as Data Augmentors: The AC-GAN achieved the highest sentence-level performance (F1-score of 52.34%), proving that synthetic data generation can improve model generalization in low-resource legal domains.
- Correlation: A strong positive correlation ($r = 0.57$) exists between the density of vague words and the overall perceived vagueness of a sentence.
Conclusion and Future Work
While the models show promise, challenges remain, particularly with class imbalance—extremely vague sentences are rare (1.8% of the data) and thus harder to learn. Future research could explore sequence-to-sequence models to "translate" vague legalise into clearer, actionable language, further empowering users to understand their digital rights.
Summary updated