Studying the Online Deepfake Community

Brian Timmerman, Pulak Mehta, Progga Deb, Kevin Gallagher, Brendan Dolan-Gavitt, Siddharth Garg, Rachel GreenstadtJournal of Online Trust and Safety 2023 · DOI

TL;DR

This paper dives into the online deepfake community, analyzing the now-defunct r/deepfakes subreddit and the current dominant platform, MrDeepFakes (MDF). It reveals a complex ecosystem where technical discussions coexist with a significant demand for nonconsensual pornographic deepfakes, and highlights how early deplatforming has fostered distrust in alternative platforms.

What Problem Does This Paper Address?

The rapid advancement of deepfake technology presents a dual-use dilemma: it offers creative applications in art and industry but also poses serious risks for disinformation, identity fraud, and harassment. As platforms have cracked down on deepfake content, communities have migrated, leading to the rise of dedicated hubs like MrDeepFakes. This paper addresses the question of how these communities function, what their members' motivations and perceptions are, and how the historical context of deplatforming has shaped their current landscape, particularly concerning the prevalence of nonconsensual content and distrust of external entities.

Methods

The study employs a mixed-methods approach, combining quantitative and qualitative analysis.

  • Quantitative Analysis: This involved collecting and analyzing platform usage statistics, including thread counts, post counts, user engagement, and the activity levels of different discussion categories. Data was scraped from MDF and obtained via the Pushshift API for r/deepfakes.
  • Qualitative Analysis: Thematic analysis was conducted on a subset of MDF's "Discussion" sub-forum threads to understand user sentiment, ethical arguments, and perceptions of legality and societal views. This involved manual coding and interpretation of conversations.

Datasets

  • r/deepfakes: Archived posts and comments from the subreddit, collected via the Pushshift API, spanning from December 2017 to February 2018.
  • MrDeepFakes (MDF): Data collected via a custom web scraper, focusing on forum posts from February 2018 to August 2021. This included thread content, author information, and post metadata.
  • YouTube Deepfake Tutorials: A sample of 30 YouTube tutorials related to deepfakes was analyzed to understand how users are directed to MDF.

Key Insights and Ideas

  • Dual Nature of MDF: MDF serves a dual purpose: a hub for technical deepfake knowledge and a marketplace for nonconsensual pornographic deepfakes. This duality is a direct consequence of deplatforming, forcing users seeking technical knowledge onto a platform where illicit content is dominant.
  • Distrust Fuels Isolation: The history of deplatforming has fostered a deep-seated distrust of external platforms and media among MDF users. This leads to a self-imposed isolation, where members view MDF as the only safe space, reinforcing the community's insular nature.
  • Technicality as a Gateway: While the primary demand on MDF is for NSFW content, technical discussions and resource sharing are prevalent. This suggests that the technical aspects of deepfakes act as a gateway for new users, who are then exposed to the platform's dominant market.
  • Celebrities as Primary Targets: The "Celebrity Facesets" sub-forum reveals a strong preference for targeting women in entertainment, likely driven by market demand. However, a significant portion of targets also pose risks for disinformation.

Key Results

  • Activity Levels: r/deepfakes experienced a peak in early 2018, with MDF stabilizing around 724.5 posts per month from 2020-2021, approximately 9.5% of r/deepfakes' peak activity.
  • Dominant Discussion Category: Technical discussions (software, hardware, assistance) are the most common category on both platforms, accounting for 49.2% of r/deepfakes threads and 42% of MDF threads.
  • User Engagement: Most users exhibit shallow engagement, typically posting in only one topic domain (77.8% on MDF, 93.6% on r/deepfakes) and making few posts overall (92.7% on MDF, 95.0% on r/deepfakes making 1-10 posts).
  • Market Demand: 98.5% of content requests on MDF are for NSFW videos, indicating a strong demand for nonconsensual pornographic deepfakes.
  • Target Demographics: Deepfake targets are predominantly female (68.8%) and from English-speaking countries (50.3% American). Entertainment figures are the most common targets (77.7%), but 22.3% have potential for disinformation.

Limitations and Future Work

  • Data Availability: The archival data for r/deepfakes is incomplete, with comments from the final month missing. MDF data also excludes personally identifying information.
  • Interview Response Rate: The study experienced a low response rate for paid interviews, limiting the depth of qualitative insights from direct user interaction.
  • Emerging Market: The paper acknowledges a complex, emerging deepfake market that largely operates outside public view, suggesting future work to explore its scope and costs.
  • Technical Assistance Motivation: Further research is needed to understand why MDF remains a preferred hub for technical assistance despite its association with illicit content.
  • Dialogue Facilitation: Additional efforts are recommended to facilitate dialogue with community members, as outreach attempts in this study received limited engagement.

Summary updated

Abstract

Deepfakes have become a dual-use technology with applications in the domains of art, science, and industry. However, the technology can also be leveraged maliciously in areas such as disinformation, identity fraud, and harassment. In response to the technology's dangerous potential many deepfake creation communities have been deplatformed, including the technology's originating community – r/deepfakes. Opening in February 2018, just eight days after the removal of r/deepfakes, MrDeepFakes (MDF) went online as a privately owned platform to fulfill the role of community hub, and has since grown into the largest dedicated deepfake creation and discussion platform currently online. This position of community hub is balanced against the site's other main purpose, which is the hosting of deepfake pornography depicting public figures- produced without consent. In this paper we explore the two largest deepfake communities that have existed via a mixed methods approach utilizing quantitative and qualitative analysis. We seek to identify how these platforms were and are used by their members, what opinions these deepfakers hold about the technology and how it is seen by society at large, and identify how deepfakes-as-disinformation is viewed by the community. We find that there is a large emphasis on technical discussion on these platforms, intermixed with potentially malicious content. Additionally, we find the deplatforming of deepfake communities early in the technology's life has significantly impacted trust regarding alternative community platforms.
BibTeX
@article{timmerman2023studying,
  author = {Brian Timmerman and Pulak Mehta and Progga Deb and Kevin Gallagher and Brendan Dolan-Gavitt and Siddharth Garg and Rachel Greenstadt},
  title = {{Studying the Online Deepfake Community}},
  journal = {Journal of Online Trust and Safety},
  year = {2023},
  doi = {10.54501/jots.v2i1.126},
}
Figure