VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Web

Andrei Manolache, Florin Brad, Antonio Barbalau, Radu Tudor Ionescu, Marius Popescu

Abstract

The Dark Web represents a hotbed for illicit activity, where users communicate on different market forums in order to exchange goods and services. Law enforcement agencies benefit from forensic tools that perform authorship analysis, in order to identify and profile users based on their textual content. However, authorship analysis has been traditionally studied using corpora featuring literary texts such as fragments from novels or fan fiction, which may not be suitable in a cybercrime context. Moreover, the few works that employ authorship analysis tools for cybercrime prevention usually employ ad-hoc experimental setups and datasets. To address these issues, we release VeriDark: a benchmark comprised of three large-scale authorship verification datasets and one authorship identification dataset obtained from user activity from either Dark Web-related Reddit communities or popular illicit Dark Web market forums. We evaluate competitive NLP baselines on the three datasets and perform an analysis of the predictions to better understand the limitations of such approaches.

Introduction

Authorship Verification (AV)—the task of determining if two documents were written by the same individual—is a cornerstone of digital forensics. In the context of the Dark Web, where anonymity is the default and illicit activities are rampant, the ability to link accounts across different marketplaces or forums is invaluable for law enforcement.

Historically, AV research has relied on datasets like PAN, which consist of literary works, novels, and fan fiction. However, the linguistic landscape of the Dark Web is vastly different. Communication is characterized by short, informal posts, heavy use of slang, PGP signatures, and domain-specific jargon such as drug names or hacking terminology. This "domain shift" means that models trained on clean, literary prose often fail when applied to the messy reality of illicit digital spaces. VeriDark addresses this gap by providing a massive, curated benchmark reflecting actual communication patterns in the Dark Web.

The VeriDark Benchmark

VeriDark is the largest AV benchmark to date, aggregating data from three primary sources:

  • Agora: Data from one of the most popular historical Dark Web marketplaces, containing approximately 4.2 million training pairs.
  • SilkRoad1: Data from the original, infamous Silk Road marketplace, comprising roughly 614,000 pairs.
  • DarkReddit+: Comments from the now-defunct /r/darknetmarkets subreddit, providing approximately 120,000 pairs.

Compared to traditional benchmarks, VeriDark samples are significantly shorter (averaging ~120 words vs. ~3,900 in PAN) and more numerous, presenting a unique challenge for modern NLP models.

Methodology

We employed a supervised learning approach using BERT-based models fine-tuned as binary classifiers. To address the 512-token limit of BERT and the varying lengths of forum posts, we implemented a chunking strategy:

  1. Longer documents are split into chunks of 254 tokens.
  2. The model evaluates pairs of chunks from Document A and Document B.
  3. The final authorship probability is calculated as the average of the probabilities across all chunk pairs.

Beyond verification, we also explored Authorship Identification (AI). In a realistic forensic scenario, a post might not belong to any known suspect. To simulate this "open-world" problem, we introduced an "Other" class, testing the model's ability to distinguish a specific user's style from the general background noise of the marketplace.

Key Insights and Results

The Domain Gap

Our experiments confirmed that stylometry is highly context-dependent. Models trained on the literary PAN dataset achieved an average score of only ~68% on Dark Web data. However, when trained on VeriDark itself, performance jumped to ~87%. This underscores the necessity of domain-specific training data for cybersecurity applications.

Spurious Features and the "Drug Name Trap"

Qualitative analysis revealed that models often rely on "spurious features"—shared named entities like specific drug names or vendor handles—to make "Same Author" predictions. While this can boost accuracy in specific contexts, it risks false positives when different users are simply discussing the same illicit products. This suggests that current models may be learning what is being discussed rather than how it is being written.

Cross-Marketplace Transfer

Interestingly, models trained on one marketplace (e.g., Agora) performed remarkably well on others (e.g., SilkRoad1). This suggests the existence of a universal "Dark Web register" or subcultural style that transcends individual platforms, allowing for the development of more robust, general-purpose forensic tools.

Performance Metrics

The BERT-based approach significantly outperformed traditional non-neural compression-based baselines. On the Agora test set, the model achieved an AUC of 93.5% and an F1-score of 84.9%. Aggregating all VeriDark datasets for training produced the most robust results, with an average metric of 87.1% across all test sets.

Limitations and Future Work

While VeriDark provides a powerful foundation, several real-world challenges remain:

  • Sybils and Shared Accounts: The ground truth is complicated by "Sybils" (one person using multiple accounts) and shared accounts (multiple people using one vendor profile).
  • Entity Masking: Future research should focus on masking named entities to ensure models focus strictly on stylistic signatures.
  • Multimodal Forensics: Incorporating non-textual data, such as product images or PGP key metadata, could further enhance the accuracy of authorship analysis in illicit markets.

VeriDark is publicly available to the research community to foster the development of more effective tools for cybercrime prevention and digital forensics.

Summary updated

Abstract

The DarkWeb represents a hotbed for illicit activity, where users communicate on different market forums in order to exchange goods and services. Law enforcement agencies benefit from forensic tools that perform authorship analysis, in order to identify and profile users based on their textual content. However, authorship analysis has been traditionally studied using corpora featuring literary texts such as fragments from novels or fan fiction, which may not be suitable in a cybercrime context. Moreover, the few works that employ authorship analysis tools for cybercrime prevention usually employ ad-hoc experimental setups and datasets. To address these issues, we release VeriDark: a benchmark comprised of three large scale authorship verification datasets and one authorship identification dataset obtained from user activity from either Dark Web related Reddit communities or popular illicit Dark Web market forums. We evaluate competitive NLP baselines on the three datasets and perform an analysis of the predictions to better understand the limitations of such approaches. We make the datasets and baselines publicly available at https://github.com/bit-ml/VeriDark
BibTeX
@inproceedings{manolache2022veridark,
  author = {Andrei Manolache and Florin Brad and Antonio Barbalau and Radu Tudor Ionescu and Marius Popescu},
  title = {{VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Web}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=rA4wbNIfSVU},
}
Figure