TL;DR
This paper dives into the online deepfake community, analyzing the now-defunct r/deepfakes subreddit and the current dominant platform, MrDeepFakes (MDF). It reveals a complex ecosystem where technical discussions coexist with a significant demand for nonconsensual pornographic deepfakes, and highlights how early deplatforming has fostered distrust in alternative platforms.
What Problem Does This Paper Address?
The rapid advancement of deepfake technology presents a dual-use dilemma: it offers creative applications in art and industry but also poses serious risks for disinformation, identity fraud, and harassment. As platforms have cracked down on deepfake content, communities have migrated, leading to the rise of dedicated hubs like MrDeepFakes. This paper addresses the question of how these communities function, what their members' motivations and perceptions are, and how the historical context of deplatforming has shaped their current landscape, particularly concerning the prevalence of nonconsensual content and distrust of external entities.
Methods
The study employs a mixed-methods approach, combining quantitative and qualitative analysis.
- Quantitative Analysis: This involved collecting and analyzing platform usage statistics, including thread counts, post counts, user engagement, and the activity levels of different discussion categories. Data was scraped from MDF and obtained via the Pushshift API for r/deepfakes.
- Qualitative Analysis: Thematic analysis was conducted on a subset of MDF's "Discussion" sub-forum threads to understand user sentiment, ethical arguments, and perceptions of legality and societal views. This involved manual coding and interpretation of conversations.
Datasets
- r/deepfakes: Archived posts and comments from the subreddit, collected via the Pushshift API, spanning from December 2017 to February 2018.
- MrDeepFakes (MDF): Data collected via a custom web scraper, focusing on forum posts from February 2018 to August 2021. This included thread content, author information, and post metadata.
- YouTube Deepfake Tutorials: A sample of 30 YouTube tutorials related to deepfakes was analyzed to understand how users are directed to MDF.
Key Insights and Ideas
- Dual Nature of MDF: MDF serves a dual purpose: a hub for technical deepfake knowledge and a marketplace for nonconsensual pornographic deepfakes. This duality is a direct consequence of deplatforming, forcing users seeking technical knowledge onto a platform where illicit content is dominant.
- Distrust Fuels Isolation: The history of deplatforming has fostered a deep-seated distrust of external platforms and media among MDF users. This leads to a self-imposed isolation, where members view MDF as the only safe space, reinforcing the community's insular nature.
- Technicality as a Gateway: While the primary demand on MDF is for NSFW content, technical discussions and resource sharing are prevalent. This suggests that the technical aspects of deepfakes act as a gateway for new users, who are then exposed to the platform's dominant market.
- Celebrities as Primary Targets: The "Celebrity Facesets" sub-forum reveals a strong preference for targeting women in entertainment, likely driven by market demand. However, a significant portion of targets also pose risks for disinformation.
Key Results
- Activity Levels: r/deepfakes experienced a peak in early 2018, with MDF stabilizing around 724.5 posts per month from 2020-2021, approximately 9.5% of r/deepfakes' peak activity.
- Dominant Discussion Category: Technical discussions (software, hardware, assistance) are the most common category on both platforms, accounting for 49.2% of r/deepfakes threads and 42% of MDF threads.
- User Engagement: Most users exhibit shallow engagement, typically posting in only one topic domain (77.8% on MDF, 93.6% on r/deepfakes) and making few posts overall (92.7% on MDF, 95.0% on r/deepfakes making 1-10 posts).
- Market Demand: 98.5% of content requests on MDF are for NSFW videos, indicating a strong demand for nonconsensual pornographic deepfakes.
- Target Demographics: Deepfake targets are predominantly female (68.8%) and from English-speaking countries (50.3% American). Entertainment figures are the most common targets (77.7%), but 22.3% have potential for disinformation.
Limitations and Future Work
- Data Availability: The archival data for r/deepfakes is incomplete, with comments from the final month missing. MDF data also excludes personally identifying information.
- Interview Response Rate: The study experienced a low response rate for paid interviews, limiting the depth of qualitative insights from direct user interaction.
- Emerging Market: The paper acknowledges a complex, emerging deepfake market that largely operates outside public view, suggesting future work to explore its scope and costs.
- Technical Assistance Motivation: Further research is needed to understand why MDF remains a preferred hub for technical assistance despite its association with illicit content.
- Dialogue Facilitation: Additional efforts are recommended to facilitate dialogue with community members, as outreach attempts in this study received limited engagement.
Summary updated