GreaseVision: Rewriting the Rules of the Interface

Siddhartha Datta, Konrad Kollnig, Nigel Shadbolt

TL;DR

GreaseVision is a developer-less, low-code framework that empowers end-users to collaboratively combat digital harms—like dark patterns and hate speech—by modifying user interfaces in real-time. By combining screenome visualization with few-shot machine learning and visual overlays, it bridges the gap between identifying digital harms and deploying immediate, personalized interventions.

What Problem Does This Paper Address?

Digital harms are highly individualized and evolve rapidly. Whether it's a manipulative "dark pattern" designed to trick you into a subscription, an addictive recommendation feed, or toxic hate speech, what harms one user might not affect another.

Traditional software development cycles and legal remedies are too slow and rigid to protect users effectively. Furthermore, existing tools separate harm analysis (like tracking screen time) from intervention deployment (like ad blockers). There is a distinct lack of flexible, platform-agnostic tools that allow non-technical users to detect harms in their daily digital lives and immediately fix them without needing root access or programming knowledge.

Methods

GreaseVision introduces a server-client architecture where user interfaces (desktop or mobile) run inside emulated environments (Docker containers or virtual machines) on a central server.

Architecture of GreaseTerminator vs GreaseVision

Key components include:

  • Visual Overlay Modifications: Instead of altering an application's source code, GreaseVision captures screen images, processes them, and renders a visual overlay (e.g., blocking out metrics or blurring text) before streaming the final image back to the user's browser. This server-side processing eliminates the overlay-underlay latency mismatch common in client-side tools.
  • Screenome Visualization & HITL: Users can review their "screenome"—a time-series sequence of their actual screen history—and use a graphical interface to draw bounding boxes around harmful elements. This Human-in-the-Loop (HITL) approach turns raw usage data into training data.
Screenome visualization page
  • Intervention Hooks:
  • Text Hook: Uses EAST text detection and Tesseract OCR to find and censor specific text.
  • Mask Hook: Employs multi-scale multi-template matching to identify and block recurring GUI elements (like "Like" counts or recommendation bars).
  • Model Hook: Integrates deep learning models (e.g., RoBERTa) that can be fine-tuned to detect complex harms like hate speech.
  • Collaborative Training: Interventions and annotated data can be shared across a network of users, allowing models to be fine-tuned collectively.

Datasets

  • Dynamically Generated Hate Speech Dataset: Used to evaluate the collaborative fine-tuning of text models. The authors focused specifically on the subset of hate speech targeted at women (1,652 training sentences, 187 test sentences), balanced against 15,184 non-hate sentences.

Key Insights and Ideas

  • Shifting Maintenance to the End-User: GreaseVision flips the traditional software lifecycle on its head. Instead of waiting for platform developers to fix UI flaws or remove toxic features, users take over the maintenance and repair of their own software interfaces.
  • The Power of Illusory Modifications: You don't need to rewrite an app's backend code to change its impact. Altering the visual perception of an interface directly changes user behavior and mitigates psychological harms (e.g., removing engagement metrics to curb social media validation loops).
  • Network Effects in AI Training: A single user trying to train a hate-speech detector faces a massive data bottleneck. By crowdsourcing annotations through screenome history, a small network of users can rapidly train robust, niche ML models without professional developer oversight.

Key Results

  • Low Latency: GreaseVision achieves a theoretical processing and streaming latency of ~5.03ms (benchmarked with an NVIDIA GeForce RTX 2080), which is fast enough to prevent visual lag or overlay mismatch during live browsing.
  • Cross-Platform UI Occlusion: The system successfully demonstrated one-shot element removal (e.g., blocking the "Stories" bar on Instagram/LinkedIn or metrics on YouTube) across native Android, iOS, mobile browsers, and desktop browsers using a single framework.
  • Accelerated Model Convergence: In hate speech detection simulations, collaborative fine-tuning dramatically accelerated model accuracy. When multiple users contributed data, the fine-tuned RoBERTa model converged toward baseline accuracy much faster than a single user training a model alone.
Convergence of few-shot/fine-tuned models

Limitations and Future Work

  • Hardware/OS Access: Because it relies on emulators, the current implementation lacks access to device-native features like the camera, audio, and haptics.
  • Data Standardization: Real-world crowd-sourced labeling can be messy and non-standardized. The authors note that future iterations will require algorithmic safeguards like out-of-distribution detection or ensemble models to handle noisy user annotations.
  • The Risk of Malevolent Use: Maximizing user autonomy means users can also choose to filter out beneficial information, potentially trapping themselves in harmful echo chambers or radicalizing bubbles (e.g., completely blocking content related to women's rights or health). Future work must carefully consider how to regulate these personalized digital realities.

Summary updated

Abstract

Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. As a result, we still lack a systematic approach to study harms and produce interventions for end-users. We put forward GreaseVision, a new framework that enables end-users to collaboratively develop interventions against harms in software using a no-code approach and recent advances in few-shot machine learning. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of harms and interventions at scale.
BibTeX
@inproceedings{datta2022greasevision,
  author = {Siddhartha Datta and Konrad Kollnig and Nigel Shadbolt},
  title = {{GreaseVision: Rewriting the Rules of the Interface}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure