TL;DR
This paper investigates the automatic detection of legal violations in cookie banner texts, a crucial step for ensuring user privacy online. The authors evaluate several state-of-the-art language models, finding that while BERT and LEGAL-BERT show promise, the small and imbalanced dataset limits their overall performance, highlighting the need for more annotated data.
What Problem Does This Paper Address?
Navigating the web often involves encountering cookie banners, which are designed to obtain user consent for personal data processing. However, research indicates that a significant majority of these banners violate regulations like the GDPR and ePrivacy Directive. These violations often stem from the language used, employing tactics like vagueness, framing, and technical jargon to subtly influence user decisions. Manually auditing these banners is a monumental task for regulators. This paper addresses the challenge of automatically identifying these legally problematic textual patterns in cookie banners to aid in auditing and enforcement efforts.
Methods
The researchers employed a comparative study of several advanced natural language processing (NLP) models for classifying legal violations in cookie banner texts. The models tested include:
- BERT: A foundational Transformer-based model, fine-tuned for this task.
- BERT with LIWC features: BERT embeddings concatenated with features derived from the Linguistic Inquiry and Word Count (LIWC) dictionary, aiming to capture stylistic and psychological aspects of the text.
- LEGAL-BERT: A BERT model pre-trained on a large corpus of English legal texts, hypothesized to perform better on legally-tinged language.
- BART in a Zero-Shot setting: The BART model was used without specific fine-tuning for this task, reframing the classification as a Natural Language Inference (NLI) problem.
The classification was performed on a per-class basis, and the models were evaluated based on their accuracy.
Datasets
The study utilized a dataset of 407 cookie banner text segments, originally annotated by five experts for legal violations according to GDPR and ePrivacy Directive requirements. This dataset was derived from the work of Santos et al. (2021). The authors note that the original dataset was annotated segment-wise, but for this study, they aimed to label the entire cookie banner as a whole. Due to data sparseness, some original annotation classes were omitted, resulting in five primary classes: Consent options presence, Misleading language, Framing, Purpose, and Technical jargon. The dataset is described as small and having unbalanced class distributions.
Key Insights and Ideas
- No single model reigns supreme: The study found that no single model consistently outperformed all others across all detected violation categories. This suggests that different models might be better suited for specific types of legal violations.
- LEGAL-BERT shows promise but isn't a silver bullet: While LEGAL-BERT, pre-trained on legal texts, performed well, it didn't universally outperform general-purpose models like BERT. This indicates that while domain-specific pre-training is beneficial, it doesn't guarantee superiority in all nuanced legal language detection tasks.
- Data scarcity is a major bottleneck: The small size and imbalanced nature of the dataset significantly impacted model performance, particularly for minority classes. This is a critical takeaway, emphasizing that more comprehensive and balanced annotated data is essential for robust automatic detection.
- LIWC features offer marginal gains: Combining BERT with LIWC features showed some potential, particularly for capturing stylistic elements, but did not lead to a dramatic improvement across the board.
- Zero-shot BART struggles: The BART model in a zero-shot setting, while flexible, did not achieve competitive accuracy compared to fine-tuned models, suggesting that explicit training on this specific task is more effective.
Key Results
- Overall Accuracy: BERT and LEGAL-BERT generally provided the highest accuracy results, ranging from 70% to 97% for certain classes.
- Class-Specific Performance:
- Consent options presence: BART achieved the highest accuracy (91.65%).
- Purpose: LEGAL-BERT led with 93.4% accuracy.
- Technical jargon: LEGAL-BERT performed best with 81.3% accuracy.
- Misleading language and Framing: These classes exhibited the lowest accuracy, with percentages dropping to around 60% for some models. This is attributed to the complexity of these categories, which often rely on stylistic nuances and have multiple labels.
- Minority Class Challenges: LEGAL-BERT's F1 scores were lower for minority classes compared to BERT, and it struggled with misclassifying "No framing" as "Positive framing."
- BART's performance: BART generally performed the worst across most classes but was less affected by data imbalance.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Dataset Size: The primary limitation is the small size and unbalanced distribution of the manually annotated dataset. This restricts the generalizability and robustness of the models.
- Model Limitations: The study highlights that off-the-shelf or minimally fine-tuned models may not yield reliable results for auditing purposes.
Future work directions suggested include:
- Curating Larger, More Diverse Datasets: The most crucial next step is to create larger and more varied training datasets for cookie banner classification.
- Developing Efficient Annotation Pipelines: Streamlining the human-in-the-loop annotation process is vital to accelerate data collection and model development.
- Building Robust, Adaptive Algorithms: Given the evolving nature of cookie banner language and regulations, developing algorithms that can adapt to these changes is essential.
- Further Model Exploration: Investigating models specifically trained or fine-tuned for this task, potentially incorporating more sophisticated feature engineering or attention mechanisms, could yield better results.
Summary updated