VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Web

Andrei Manolache, Florin Brad, Antonio Barbalau, Radu Tudor Ionescu, Marius Popescu

Overview

VeriDark is a massive new benchmark designed to train and evaluate AI models for authorship verification in the high-stakes world of cybersecurity. By providing over 5 million text pairs from Dark Web forums and related Reddit communities, researchers offer a specialized alternative to traditional literary-based datasets, helping law enforcement better identify cybercriminals across different platforms.

The Challenge of Dark Web Authorship

Identifying a user across different online platforms is a cornerstone of digital forensics. This is known as Authorship Verification (AV): given two snippets of text, did the same person write both?

While AV has been studied for decades, most existing benchmarks rely on "clean" data like 19th-century novels or fan fiction. The Dark Web is a different beast entirely. It represents a hotbed for illicit activity where communication is defined by heavy slang, technical jargon (e.g., hacking, encryption), emojis, and a frequent disregard for formal grammar.

Current state-of-the-art models trained on literary data suffer from a severe domain shift—they simply do not know how to handle the noisy and concise nature of forum posts. Furthermore, existing cybersecurity datasets are often small or rely on ad-hoc experimental setups. VeriDark fills this gap by providing a standardized, large-scale benchmark specifically tailored for this domain.

The VeriDark Datasets

The VeriDark benchmark comprises three primary datasets, totaling over 5.4 million examples from more than 60,000 authors:

  • Agora: Scraped from the defunct Agora market forum. This is the largest dataset in the benchmark (~4.6 million pairs).
  • SilkRoad1: Sourced from the infamous Silk Road 1 marketplace (~680k pairs).
  • DarkReddit+: Comments from the /r/darknetmarkets subreddit (~120k pairs).
User activity distribution across the three VeriDark datasets.

Compared to traditional benchmarks like PAN, VeriDark's texts are much shorter—averaging around 120 words compared to PAN's ~3,900 words. However, the sheer volume of examples is orders of magnitude larger, providing a unique challenge for modern NLP models.

Word count distributions for Agora, SilkRoad1, and DarkReddit+, highlighting the concise nature of Dark Web posts.

Methodology

To establish strong baselines, the researchers fine-tuned BERT-based models as binary classifiers. Because forum posts can vary significantly in length, they employed a robust "chunking" strategy:

  1. Preprocessing: PGP signatures and keys were stripped out (as these would act as "cheat codes" for an AI) and content was filtered for English.
  2. Chunking: Each document was split into 254-token segments.
  3. Pairing: Corresponding chunks from two different documents were paired together.
  4. Aggregation: The model predicted the probability of a "same author" match for each chunk pair, averaging these probabilities to produce a final verdict for the document pair.

The team also introduced a 10-way Author Identification task to simulate a scenario where an investigator tries to match a specific comment to one of the ten most active known users.

Key Insights and Results

  • The Domain Gap is Real: Models trained on standard PAN (literary) datasets performed significantly worse on Dark Web data, confirming that cyber-slang requires specialized training.
  • Cross-Platform Transfer: Interestingly, a model trained on one Dark Web forum (like Agora) performs quite well when tested on another (like Silk Road). This suggests the existence of a universal "Dark Web register" or distinct style of writing.
  • Verification Performance: The BERT-based baselines achieved solid results, with F1 scores ranging from 75% to 85% across the different datasets. Training on the combined "All" dataset produced the most robust models, showing that more data helps the AI ignore platform-specific quirks.
  • Identification Accuracy: On the 10-way classification task, the model reached an accuracy of 84.6% for Dark Web comments.
  • Spurious Features: The authors found that models often latch onto "named entities"—like specific drug names or software like "Whonix"—to make decisions. While these are strong signals, they can be misleading if different authors are simply discussing the same popular product.

Limitations and Future Work

The authors highlight several noisy realities of the Dark Web that complicate AV:

  • Sybils: One person may use multiple accounts (pseudonyms), leading to false negatives in the training data (the model thinks they are different people, but they are the same).
  • Shared Accounts: Conversely, multiple people might share one vendor account, leading to false positives.
  • Named Entity Mitigation: Future work could involve automatically masking drug names or vendor handles to force the model to focus purely on stylometry (how someone writes) rather than content (what they are writing about).

Finally, the authors acknowledge the dual-use nature of this technology. While it aids law enforcement, it could be used by malicious actors to deanonymize whistleblowers. To mitigate this, dataset access is restricted to researchers who agree to strict ethical use terms.

Summary updated

Abstract

The Dark Web represents a hotbed for illicit activity, where users communicate on different market forums in order to exchange goods and services. Law enforcement agencies benefit from forensic tools that perform authorship analysis, in order to identify and profile users based on their textual content. However, authorship analysis has been traditionally studied using corpora featuring literary texts such as fragments from novels or fan fiction, which may not be suitable in a cybercrime context. Moreover, the few works that employ authorship analysis tools for cybercrime prevention usually employ ad-hoc experimental setups and datasets. To address these issues, we release VeriDark: a benchmark comprised of three large scale authorship verification datasets and one authorship identification dataset obtained from user activity from either Dark Web related Reddit communities or popular illicit Dark Web market forums. We evaluate competitive NLP baselines on the three datasets and perform an analysis of the predictions to better understand the limitations of such approaches. We make the datasets and baselines publicly available at https://github.com/bit-ml/VeriDark .
BibTeX
@inproceedings{manolache2022veridark,
  author = {Andrei Manolache and Florin Brad and Antonio Barbalau and Radu Tudor Ionescu and Marius Popescu},
  title = {{VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Web}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=rA4wbNIfSVU},
}
Figure