Detecting Botnets Through Log Correlation

Yousof Al-Hammadi, Uwe AickelinSsrn 2006 · DOI · OpenReview

TL;DR

This paper proposes a novel method for detecting botnets by analyzing the correlation of changes in log file sizes across multiple hosts. By focusing on synchronized behavioral shifts rather than specific traffic patterns, their approach offers a more robust and efficient way to identify malicious coordinated activity.

What Problem Does This Paper Address?

Botnets, vast networks of compromised computers, pose a significant threat through coordinated attacks like DDoS, keylogging, and data theft. Detecting these botnets is crucial, but existing methods face challenges. Monitoring network traffic on standard IRC ports can miss bots operating on non-standard ports, and analyzing traffic patterns often yields false negatives due to the inherent variability in connection lengths and data transfer sizes. Furthermore, the shift towards Peer-to-Peer (P2P) botnet architectures makes tracing the bot herder increasingly difficult. This paper aims to address the limitations of current detection techniques by focusing on a more fundamental aspect of botnet behavior: synchronized activity.

Methods

The core of the proposed method lies in log correlation. The authors intercept Windows API socket function calls (like send(), recv(), connect()) made by communication applications, such as IRC clients. These calls and their arguments are logged. A separate process then monitors the size of these log files on different hosts.

The key innovation is the correlation algorithm. It analyzes changes in log file sizes over time. If log file sizes across multiple hosts change in a synchronized manner (all increasing or all decreasing together), it indicates a high correlation. Conversely, if changes are random or uncorrelated, it suggests normal activity. The algorithm quantifies this correlation by counting "correlated values" (CV) and "uncorrelated values" (UCV). A threshold (T) is then applied to the CV to detect suspicious activity. This approach bypasses the need to inspect packet content or monitor specific ports, making it resilient to encryption and non-standard configurations.

Datasets

The authors set up a small virtual IRC network using VMWare machines running Windows XP Pro SP2. This network consisted of one IRC server and three IRC clients. They generated datasets by simulating both normal user behavior (conversation, file transfers) and botnet attack scenarios (receiving commands, flood attacks) using a bot named sdbot05b. No pre-existing public datasets were used; instead, they created their own controlled environment for experimentation.

Key Insights and Ideas

  • Focus on synchronized behavior, not content: The most significant insight is that botnets, by their nature, involve coordinated actions. This synchronization manifests as correlated changes in log file sizes across compromised machines, regardless of the specific commands or traffic patterns.
  • Resilience to encryption and non-standard ports: By monitoring log file size changes rather than packet content, the method is inherently resistant to encrypted traffic and bots operating on non-standard ports, which are common evasion tactics.
  • Distinguishing normal vs. abnormal activity: The paper demonstrates that while normal activities like file transfers can cause sudden log size changes, the correlation across multiple hosts is significantly lower compared to botnet activity. Botnets exhibit a higher degree of synchronized log file size changes.
  • The "uncorrelated" event in attack cases: Interestingly, the authors note that even in attack scenarios, there can be uncorrelated events. This is attributed to potential delays in responding to bot herder commands, which is a valuable nuance for detection.

Key Results

  • The correlation algorithm successfully distinguished between normal and botnet activity.
  • In normal scenarios (conversation, file transfers), a low correlation was observed between log file size changes across hosts.
  • In attack scenarios (receiving commands, flood attacks), a high correlation was observed, indicating synchronized activity.
  • A Receiver Operating Characteristic (ROC) analysis showed that a threshold above 70% of the log file size change was effective in detecting abnormal activity with zero false negatives and a low false positive rate. Setting the threshold below 70% increased false positives while maintaining a high detection rate.

Limitations and Future Work

  • Current focus on centralized botnets: The current approach is designed for centralized botnets. Detecting botnets in a Peer-to-Peer (P2P) network is identified as future work.
  • Distinguishing individual bots: The current method detects botnet activity at a network level. Future work aims to monitor the number of API function calls to detect individual bots within a host.
  • Amount of change not considered: The current algorithm focuses on the correlation of changes, not the magnitude of the change in log file size. This is noted as an area for future exploration.
  • Virtual environment limitations: Experiments were conducted in a virtual environment, which may not perfectly replicate real-world network dynamics.
  • False positives in normal cases: The paper acknowledges that some correlated events can occur in normal scenarios, particularly due to the synchronized recording of data every second and the overhead of virtual machine switching.

Summary updated

Abstract

Botnets, which consist of thousands of compromised machines, can cause significant threats to other systems by launching Distributed Denial of Service (SSoS) attacks, keylogging, and backdoors. In response to these threats, new effective techniques are needed to detect the presence of botnets. In this paper, we have used an interception technique to monitor Windows Application Programming Interface (API) functions calls made by communication applications and store these calls with their arguments in log files. Our algorithm detects botnets based on monitoring abnormal activity by correlating the changes in log file sizes from different hosts.
BibTeX
@misc{alhammadi2006detecting,
  author = {Yousof Al-Hammadi and Uwe Aickelin},
  title = {{Detecting Botnets Through Log Correlation}},
  year = {2006},
  eprint = {1001.2665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1001.2665},
  howpublished = {Ssrn},
}
Figure