TL;DR
This paper introduces a novel method for detecting individual bots on a system by analyzing the correlation of specific API function calls, particularly those related to keylogging. By correlating these activities within defined time windows, the authors demonstrate a more robust detection mechanism than relying on single indicators, offering a promising approach to combatting stealthy malware.
What Problem Does This Paper Address?
Bots, malicious software controlled remotely, pose a significant threat by engaging in activities like spamming and keylogging. While botnets are a common research focus, detecting individual bots, especially those employing stealthy techniques like keylogging, remains challenging. Existing detection methods often rely on network traffic analysis or signature-based approaches, which can be circumvented by encryption or novel attack vectors. Keyloggers, in particular, are difficult to detect as they operate hidden on a user's machine, collecting sensitive information without the user's knowledge, leading to identity theft. This paper aims to fill this gap by developing a host-based detection method that focuses on the behavioral patterns of a single bot exhibiting keylogging activity.
Methods
The core of this paper's contribution lies in its behavioral analysis approach. Instead of relying on network signatures, the authors propose monitoring specific Application Programming Interface (API) function calls executed by a bot. Their method focuses on three key categories of API calls: Communication Functions (e.g., socket, send), File Access Functions (e.g., CreateFile, WriteFile), and Keyboard State Functions (e.g., GetAsyncKeyState, GetKeyboardState).
The detection mechanism involves:
- API Function Call Monitoring: A "hook" program is implemented to intercept these specific API calls made by processes suspected of being bots. This hook modifies the process's Import Address Table to redirect calls to their custom monitoring function.
- Correlation Analysis: The frequency of these captured API function calls is analyzed within specified time windows. The key innovation is the use of Spearman's Rank Correlation (SRC) to measure the correlation between different behavioral patterns. For instance, they correlate keyboard state functions (indicating keylogging) with file access functions (for storing logs) and communication functions (for sending data to the botmaster).
- Thresholding: A high correlation value (SRC > 0.5) is considered indicative of suspicious activity, suggesting a potential bot. The authors explore the impact of including or excluding "idle periods" (where no relevant API calls are made) on the correlation results.
Datasets
The paper does not describe the use of pre-existing datasets for training or evaluation. Instead, the authors created their own experimental environment and datasets by:
- Setting up a virtual IRC network on VMWare.
- Using a Windows XP SP2 machine as an infected machine.
- Employing a known spybot (referred to as
spybot[5]) for experiments. - Running various experiments (E1-E5) involving different bot behaviors and user typing patterns.
Key Insights and Ideas
- Correlation is Key: A single API function call, even one strongly associated with keylogging (like
GetAsyncKeyState), might not be sufficient for definitive detection. Legitimate applications can also use these functions. The real power comes from correlating multiple behavioral indicators. - Idle Periods Can Skew Results: Including periods of inactivity (idle periods) in the correlation analysis can artificially inflate correlation values, leading to less accurate detection. Removing these idle periods generally yields more precise results.
- Typing Patterns Matter: The way a user types (long sentences vs. short sentences) can influence the observed API call patterns and their correlation, impacting detection effectiveness. Short sentences tend to show higher correlations between keylogging and file writing functions.
- Host-Based vs. Network-Based: This work emphasizes host-based detection by monitoring internal system calls, offering a complementary approach to traditional network-based bot detection.
Key Results
The experiments demonstrated varying levels of detection confidence based on the bot's activity and the user's interaction:
- Experiment E1 (Idle Bot): Showed minimal API calls except for communication functions, making detection difficult.
- Experiment E2 (Bot Receives Commands): Revealed increased outgoing traffic and some file access functions, but no
GetAsyncKeyStatecalls, indicating a "weak detection" scenario. - Experiment E3 (Keylogger Active, No Data Sent):
- E3.1 (Long Sentences): Detected keylogging activity but showed low correlation between
GetAsyncKeyStateandWriteFile, resulting in "weak detection." - E3.2 (Short Sentences): Showed high correlation between
GetAsyncKeyStateandWriteFile, leading to "normal detection" as data wasn't sent out.
- E3.1 (Long Sentences): Detected keylogging activity but showed low correlation between
- Experiment E4 (Keylogger Active, Data Sent):
- E4.1 (Long Sentences): Similar to E3.1, with low correlation between outgoing traffic and
GetAsyncKeyState.
- E4.1 (Long Sentences): Similar to E3.1, with low correlation between outgoing traffic and
- E4.2 (Short Sentences): This was the most successful scenario, exhibiting high correlations across multiple data sets (
GetAsyncKeyStatewith outgoing traffic andWriteFile), leading to "strong detection." - Experiment E5 (mIRC Client): The normal application did not trigger any keylogging-related API calls, confirming the specificity of the detection method.
Table I summarizes these findings, showing "Strong detection" in E4.2 and "Normal detection" in E3.2, while other scenarios resulted in "Weak detection" or "No detection."
Limitations and Future Work
The authors acknowledge several limitations:
- Time Window Sensitivity: The effectiveness of the detection algorithm is highly dependent on the chosen time window size and how it aligns with different bot activities (typing, saving, sending). Mismatched time windows can lead to inaccurate correlations.
- Evasion by Random Delays: Bots could potentially evade detection by introducing random delays between actions, requiring the detection system to adapt its time window accordingly, which might negatively impact detection speed.
- Focus on Keylogging: The current work specifically targets keylogging bots. Future work could involve combining this with detection of other bot activities (e.g., SYN attacks, UDP attacks) to create a more comprehensive detection system.
- Correlation Scheme Improvement: The authors suggest that a more intelligent correlation scheme is needed, potentially moving beyond simple Spearman's Rank Correlation.
For future work, they propose using an Artificial Immune System approach to detect Peer-to-Peer (P2P) bots, indicating a direction towards more advanced and adaptive detection techniques.
Summary updated