Spamdoop: A Privacy-Preserving Big Data Platform For Collaborative Spam Detection

Ernesto Damiani, Abdelrahman AlMahmoud, Hadi Otrok, Yousof Al-HammadiZenodo 2017 · DOI

TL;DR

This paper introduces Spamdoop, a novel Big Data platform for collaborative spam detection that prioritizes both privacy and scalability. By employing a unique two-stage encoding process and leveraging MapReduce, Spamdoop can efficiently detect spam campaigns in near real-time without compromising user data, addressing a critical gap in current collaborative spam detection methods.

What Problem Does This Paper Address?

The proliferation of spam emails, often used to distribute malware, phishing links, and other malicious content, remains a significant cybersecurity challenge. Collaborative spam detection, where multiple entities share information to identify spam, offers a promising solution for handling the sheer volume of emails. However, existing collaborative methods face a critical dilemma: they either require participants to disclose the content of their emails, thus violating privacy, or they rely on distance-preserving hashing techniques that, while preserving privacy, are not scalable enough for the massive datasets involved in modern spam campaigns. This paper tackles this scalability-privacy trade-off, aiming to build a system that can effectively detect spam campaigns collaboratively without compromising user privacy.

Methods

Spamdoop employs a two-stage encoding process designed to be resilient to spam modification techniques and privacy-preserving.

  • Stage 1: Two-Stage Encoder: This stage focuses on creating a normalized representation of emails that ensures similar emails (originating from the same template) produce identical encodings. It begins by building a language model of trigrams within the email and their occurrence counts. These counts are then mapped to "channels" based on their range, effectively creating a representation that is robust to minor modifications like "hash busting." This stage is designed to be many-to-one, meaning different inputs can map to the same output.
  • Stage 2: Privacy-Preserving Hashing: To ensure non-reversibility and further compress the representation, a one-way cryptographic hash function (like SHA or CRC32) is applied to the output of Stage 1. This makes it computationally infeasible to reconstruct the original email content.

The encoded digests are then processed by:

  • Parallel Classifier: This component routes similar digests to the same "bucket" using a simple modulo operation based on the group ID. This grouping is crucial for efficient processing and storage on distributed nodes.
  • Anomaly Detector: This component identifies spam by analyzing the occurrence density of digests. It uses a histogram-based approach to detect unusual patterns, such as a sudden surge in specific digests, which are indicative of spam campaigns. The detector considers both current occurrences and historical data to identify emerging threats.

Spamdoop is built on top of a standard MapReduce framework, allowing it to scale horizontally by adding more processing nodes.

Datasets

The paper evaluates Spamdoop using a "huge synthetic spam base." Specifically, they created a dataset of 43,176,780 digests. This dataset was generated from five distinct parent spam templates, each used to create 1500 unique child spam emails. These child emails were modified with 3 to 8 word substitutions and hash busters to mimic real-world spam tactics. Additionally, one thousand non-spam emails from the CSDMC2010 SPAM corpus were used to test for false positives.

Key Insights and Ideas

  • The "Many-to-One" Encoding is Key: The core innovation is an encoding that maps similar emails (from the same template) to identical representations. This is crucial for grouping spam campaigns effectively, unlike traditional hashing which might produce slightly different hashes for similar emails.
  • Scalability via MapReduce and Bucketing: By leveraging MapReduce and a simple bucketing mechanism, Spamdoop can process massive amounts of data in parallel, overcoming the scalability limitations of previous distance-preserving hashing methods.
  • Privacy without Sacrificing Core Functionality: The two-stage encoding ensures that even though the first stage is not fully privacy-preserving, the second stage's cryptographic hash makes the overall process non-reversible. This allows for collaborative detection without exposing sensitive email content.
  • Anomaly Detection on Encoded Data: The anomaly detection mechanism works on the encoded digests, effectively grouping variations of the same spam campaign under a single digest, making detection more robust.

Key Results

  • Efficient Spam Detection: Spamdoop demonstrated the ability to process a large volume of emails (over 43 million digests) in a matter of minutes, significantly faster than the typical duration of a spam campaign (12 hours to 8 days).
  • Scalability Improvements: Adding more working nodes (slaves) in the MapReduce cluster significantly reduced processing time. For example, using 4 slaves reduced batch processing time by up to 52% compared to a single slave.
  • Performance vs. Hash Choice: Using CRC32 as the second-stage hash resulted in faster processing times compared to SHA512, though SHA512 offers stronger security guarantees.
  • Low False Positives: The second set of functionality tests, using non-spam emails, showed that the median group size was 1, and the largest group contained only 91 emails, indicating a low rate of false positives.
  • Resilience to Modifications: The encoding scheme proved effective in grouping spam emails generated from the same template, even with modifications like word substitutions and hash busters.

Limitations and Future Work

  • Accuracy vs. Performance Trade-off: The authors acknowledge that the first stage of their encoding sacrifices a small degree of accuracy for improved performance.
  • Choice of Hash Function: While CRC32 is faster, SHA512 offers better non-reversibility. The choice depends on the specific security and performance requirements.
  • Future Work: The authors suggest exploring the application of their encoding to binary data (like images) and investigating different strategies for channel size selection to further optimize performance and detection. They also note that the anomaly detection framework is agnostic to the specific technique used and could be enhanced with more advanced methods.

Summary updated

Abstract

Spam has become the platform of choice used by cyber-criminals to spread malicious payloads such as viruses and trojans. In this paper, we consider the problem of early detection of spam campaigns. Collaborative spam detection techniques can deal with large scale email data contributed by multiple sources; however, they have the well-known problem of requiring disclosure of email content. Distance-preserving hashes are one of the common solutions used for preserving privacy of email content while enabling message classification for spam detection. However, distance-preserving hashes are not scalable, thus making large scale collaborative solutions difficult to implement. As a solution, we propose Spamdoop, a Big Data privacy-preserving collaborative spam detection platform built on top of a standard Map Reduce facility. Spamdoop uses a highly parallel encoding technique that enables the detection of spam campaigns in competitive times. We evaluate our system's performance using a huge synthetic spam base and show that our technique performs favorably against the creation and delivery overhead of current spam generation tools.
BibTeX
@inproceedings{damiani2017spamdoop,
  author = {Ernesto Damiani and Abdelrahman AlMahmoud and Hadi Otrok and Yousof Al-Hammadi},
  title = {{Spamdoop: A Privacy-Preserving Big Data Platform For Collaborative Spam Detection}},
  booktitle = {Zenodo},
  year = {2017},
  doi = {10.5281/zenodo.887650},
}
Figure