Abstract
Pinterest relies on query recommendations to enhance user discovery, but these automated systems can inadvertently suggest unsafe content. This paper presents PinSets, a query-set expansion system that identifies unsafe terms by leveraging both textual features and user behavioral signals. Starting from a minimal set of just 20 seed queries, PinSets expanded the safety lexicon to over 15,000 terms with 99.3% precision. Deployment of the system resulted in a 90% reduction in unsafe query suggestions while maintaining platform engagement.
Introduction: The Challenge of Query Safety
Search engines use recommendation products like "Refinements" and "Related Topics" to help users explore interests. However, these automated suggestions can surface unsafe content, such as illegal drugs, hate speech, or adult content. Filtering these terms is notoriously difficult due to three main factors:
- Linguistic Diversity: There are hundreds of ways to refer to a single topic (e.g., slang names for marijuana like ganja or weed).
- Contextual Ambiguity: Words like "nude" or "grass" are safe in some contexts (lipstick, gardening) but unsafe in others.
- The "Tail" Problem: Most unsafe queries are rare, making it difficult for traditional machine learning models to gather enough training data.
The PinSets Methodology
The authors propose PinSets, which treats query safety as a "set expansion" problem. Instead of classifying every query from scratch, the system uses a two-phase hybrid approach:
- Phase 1 (Head/Torso Queries): The system builds a weighted bipartite graph connecting queries to their constituent words (ngrams). It uses a modified Point-wise Mutual Information (PMI) metric to identify strong associations while avoiding rare "fluke" occurrences.
- Phase 2 (Tail Queries): This phase leverages "topic stickiness"—the tendency of users to stay on a single topic during a session. If a user searches for multiple known unsafe terms, an unknown term in that same session is flagged as likely unsafe, regardless of its text.
In production, Pinterest combines this behavioral data with a deep-learned textual model (fastText). The textual model provides generalization for new queries, while the behavioral data acts as a high-precision lookup for ambiguous or non-compositional terms.
Results and Impact
Using only 20 drug-related seed queries, PinSets generated 15,670 positive examples and 1.7 million negative examples. Human evaluation confirmed a precision of 99.3%.
When deployed at Pinterest, the system achieved a 90% reduction in unsafe query suggestions. Crucially, this sanitization had a negligible impact on user engagement (less than 1% change), indicating that the system was surgical enough to avoid over-blocking safe content.
Conclusion and Future Work
PinSets demonstrates that behavioral signals can effectively overcome the limitations of text-only safety models by treating queries as "black boxes." While the current implementation is optimized for English, the behavioral logic is inherently language-agnostic. Future work includes extending the system to non-whitespace-delimited languages and applying the set expansion technique to broader taxonomies like user interests and shopping categories.
Summary updated