Surfacing Privacy Settings Using Semantic Matching

Rishabh Khandelwal, Asmit Nayak, Yao Yao, Kassem Fawaz2020 · DOI

Abstract

Online services utilize privacy settings to provide users with control over their data. However, these privacy settings are often hard to locate, causing the user to rely on provider-chosen default values. In this work, we train privacy-settings-centric encoders and leverage them to create an interface that allows users to search for privacy settings using free-form queries. In order to achieve this goal, we create a custom Semantic Similarity dataset, which consists of real user queries covering various privacy settings. We then use this dataset to fine-tune a state of the art encoder. Using this fine-tuned encoder, we perform semantic matching between the user queries and the privacy settings to retrieve the most relevant setting. Finally, we also use the encoder to generate embeddings of privacy settings from the top 100 websites and perform unsupervised clustering to learn about the online privacy settings types. We find that the most common type of privacy settings are 'Personalization' and 'Notifications', with coverage of 35.8% and 34.4%, respectively, in our dataset.

Introduction

Finding privacy settings on modern websites is often a "scavenger hunt" that discourages users from taking control of their data. Most online services offer privacy controls, but they are notoriously difficult to navigate; for instance, Facebook has over 60 different privacy settings spread across multiple URLs. This complexity leads to "dark patterns" where users simply accept provider-chosen defaults that typically favor data collection over user privacy. While previous research has focused on analyzing privacy policies, this work addresses the reachability and discoverability of the actual settings, bridging the gap between a user’s intent and specific, often obscurely named toggles.

Methodology

The researchers developed a pipeline to map natural language to privacy controls using three primary components:

Semantic Matching Architecture

They leveraged the Universal Sentence Encoder (USE) and fine-tuned it specifically for privacy-related language. The architecture uses a siamese network approach to minimize the Mean Squared Error (MSE) between the cosine similarity of embeddings and human-assigned similarity scores.

The semantic matching pipeline architecture showing the use of Universal Sentence Encoders and MSE loss for fine-tuning.

Browser Extension and Interface

A Chrome extension was built to act as a centralized interface. This tool allows users to search for settings within a single site or across multiple domains simultaneously, effectively surfacing relevant toggles regardless of the specific UI layout of the website.

The tool's interface demonstrating a cross-domain search for privacy settings across Flickr and Reddit.

Unsupervised Clustering

To understand the broader landscape of existing settings, the authors used Agglomerative Hierarchical Clustering on embeddings from the top 100 websites. This allowed them to categorize settings without predefined labels, proving more coherent than K-Means for handling the unique "outlier" settings found on specific platforms.

Datasets and Evaluation

The study utilized three manually curated datasets:

  • Semantic Similarity Dataset: 596 pairs of real-world user queries (sourced from Reddit) and actual privacy setting texts, manually scored for relevance.
  • Evaluation Dataset: An independent set of 110 query-setting pairs from 20 popular domains used to test performance.
  • Top 100 Website Crawl: Privacy setting strings extracted from the top 100 Tranco-ranked websites to analyze the current ecosystem.

Key Results and Insights

  • Model Performance: The fine-tuned USE model achieved a Top-1 accuracy of 72.9% and a Top-3 accuracy of 92.4%, significantly outperforming baseline models like standard USE, SBERT, and SRoBERTa.
  • The Privacy Landscape: Clustering revealed that the web is dominated by two types of controls: Personalization (35.8%) and Notifications (34.4%). Security-related settings (like MFA) are relatively rare in these primary privacy menus.
  • Context Dilution: Interestingly, the authors found that longer, more descriptive queries sometimes performed worse because extra information "diluted" the semantic context, leading to poorer matches with concise setting titles.

Conclusion

By transforming the search for privacy controls into a semantic matching problem, this research provides a viable path toward improving user agency online. While the current system relies on manual extraction of settings, future work will focus on automated web-scraping techniques to scale the tool across the broader web and conducting user studies to evaluate real-world impact on privacy choices.

Summary updated

Abstract

Online services utilize privacy settings to provide users with control over their data. However, these privacy settings are often hard to locate, causing the user to rely on provider-chosen default values. In this work, we train privacy-settings-centric encoders and leverage them to create an interface that allows users to search for privacy settings using free-form queries. In order to achieve this goal, we create a custom Semantic Similarity dataset, which consists of real user queries covering various privacy settings. We then use this dataset to fine-tune a state of the art encoder. Using this fine-tuned encoder, we perform semantic matching between the user queries and the privacy settings to retrieve the most relevant setting. Finally, we also use the encoder to generate embeddings of privacy settings from the top 100 websites and perform unsupervised clustering to learn about the online privacy settings types. We find that the most common type of privacy settings are 'Personalization' and 'Notifications', with coverage of 35.8% and 34.4%, respectively, in our dataset.
BibTeX
@misc{khandelwal2020surfacing,
  author = {Rishabh Khandelwal and Asmit Nayak and Yao Yao and Kassem Fawaz},
  title = {{Surfacing Privacy Settings Using Semantic Matching}},
  year = {2020},
  doi = {10.18653/v1/2020.privatenlp-1.4},
}
Figure