PrivaT5: A Generative Language Model for Privacy Policies

Mohammad Zoubi, Santosh T.y.s.s, Edgar Rosas, Matthias GrabmairACL 2024

TL;DR

This paper introduces PrivaT5, a T5-based language model specifically pre-trained on a large corpus of privacy policies. PrivaT5 demonstrates improved performance on various privacy policy-related tasks compared to the general T5 model, highlighting the benefits of domain-specific pre-training. However, it also reveals challenges generative models face with complex structured output formats, particularly in information extraction tasks.

What Problem Does This Paper Address?

In today's digital world, privacy policies are notoriously long, complex, and often filled with legal jargon, making it difficult for average users to understand them. This lack of comprehension hinders the effectiveness of privacy regulations like GDPR, which rely on users making informed choices. While Natural Language Processing (NLP) has shown promise in assisting with policy understanding, existing models haven't been specifically optimized for the unique characteristics of privacy policies, especially generative models. This paper aims to bridge this gap by developing and evaluating a generative language model tailored for the privacy policy domain.

Methods

The core of this work is the development of PrivaT5, a model built upon the T5 (Text-to-Text Transfer Transformer) architecture. PrivaT5 is created by taking pre-trained T5 models (available in small, base, and large sizes) and further pre-training them on a specialized corpus of privacy policies called PrivaSeer. This "continued pre-training" on domain-specific data is the key novelty. The authors then evaluate PrivaT5 on a suite of downstream privacy policy-related tasks, framing all tasks in a text-to-text format, which is a standard approach for T5. They also compare its performance against encoder-only models like BERT and its domain-specific variants (LegalBERT, PrivBERT) to understand the strengths and weaknesses of generative versus discriminative approaches.

Datasets

The primary dataset used for pre-training PrivaT5 is PrivaSeer, which comprises over 1 million privacy policies from nearly a million unique web domains.

For evaluation, the authors use several existing benchmarks:

  • OPP115: For multi-label classification of privacy practices.
  • PI-Extract: For extracting data-related entities (token-level multi-label classification).
  • Policy Detection: For binary classification of documents as privacy policies or not.
  • PolicyIE (IE-A and IE-B): For intent classification and slot filling related to privacy practices.
  • PrivacyQA: For question answering on privacy policies.
  • PolicyQA: For reading comprehension-style question answering, requiring text span extraction.
  • PolicySumm: For generating section titles from policy content.

Key Insights and Ideas

  • Domain-specific pre-training pays off: Continuously pre-training a general-purpose model like T5 on a specialized corpus (privacy policies) significantly boosts its performance on tasks within that domain.
  • Generative models struggle with structured output: While PrivaT5 excels in many areas, it faces challenges with tasks requiring precise, structured outputs, such as sequence tagging (like in PI-Extract and PolicyIE-B). This is where traditional encoder-only models like BERT often outperform.
  • Model scale matters, but not always linearly: Performance generally improves with larger model sizes (small, base, large), but the degree of improvement isn't always straightforward and can vary across tasks, suggesting complex interactions during fine-tuning.
  • MTL vs. STL trade-offs: Multi-task learning (MTL) can be beneficial, but it also risks "negative transfer," where learning one task can hinder performance on another, especially when tasks are dissimilar.

Key Results

  • PrivaT5 consistently outperforms the base T5 model across most evaluated privacy policy tasks, with improvements ranging from a few percentage points to significant gains in certain metrics.
  • For PI-Extract, PrivaT5-Base showed a slight decrease in performance compared to T5-Base, and for PolicyDetection, PrivaT5-Small underperformed T5-Small.
  • In sequence tagging tasks (like PI-Extract and PolicyIE-B), BERT family models (BERT, LegalBERT, PrivBERT) generally achieved superior performance compared to T5 and PrivaT5, even when T5/PrivaT5 models had significantly more parameters. For example, T5-Large (770M parameters) underperformed BERT (110M parameters) on PolicyIE-B.
  • In MTL settings, PrivaT5 generally outperformed T5, except for PolicyDetection in the base setting.

Limitations and Future Work

The authors acknowledge several limitations:

  • Corpus Bias: The PrivaSeer corpus, while large, might not fully capture the diversity of all privacy policy variations, potentially introducing biases.
  • Language Specificity: The current evaluation is limited to English-language privacy policies, restricting the model's applicability in a global context.
  • Structured Output Challenges: The paper highlights the ongoing difficulty for generative models in handling complex structured output spaces, particularly in sequence tagging.

Future work could involve:

  • Investigating the root causes of the performance gap in structured output tasks and exploring mitigation strategies.
  • Developing effective decoding mechanisms or textual transformations to improve generative model performance on information extraction.
  • Exploring the model's generalization capabilities to privacy policies in other languages.
  • Further research into the intricate dynamics of model scaling and its impact on downstream task performance.

Summary updated

BibTeX
@inproceedings{zoubi2024privat,
  author = {Mohammad Zoubi and Santosh T.Y.S.S and Edgar Rosas and Matthias Grabmair},
  title = {{PrivaT5: A Generative Language Model for Privacy Policies}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure