An Anomalous Behavior Detection Method Using System Call Sequences for Distributed Application

Chuan Ma, Limin Shen, Tao Wang, Leslie Lamport, C Din, J Dovland, E Johnsen, O Olaf, M Gul, Hussain, C Collberg, S Martin, J Myers, J Nagra, F Idrees, M Rajarajan, A Memon, J Meseguer, P Lveczky, S Forrest, S Hofmeyr, A Somayaji, T Longstaff, P Helman, J Bhangoo, W Lee, S Stolfo, D Wagner, D Dean, J Giffin, S Jha, B Miller, R Gopalakrishna, E Spafford, J Vitek, F Jianming, T Fen, W Dan, H Feng, Y Huang, M Moshirpour, A Mousavi, B Far, F Yang, T Aotain, H Masuhare, D Gupta, K Vishwanath, M Mcnett, J Tan, S Kavulya, R Gandhi, M Rohr, A Van Hoorn, W Hasselbring, R Alhajj, M Moussavi, G Milne, R Milner, J Baeten, C Hoare, J HopcroftKSII 2015 · DOI

Abstract

Distributed applications are composed of multiple nodes, which exchange information with individual nodes through message passing. Compared with traditional applications, distributed applications have more complex behavior patterns because a large number of interactions and concurrent behaviors exist among their distributed nodes. Thus, it is difficult to detect anomalous behaviors and determine the location and scope of abnormal nodes, and some attacks and misuse cannot be detected. To address this problem, we introduce a method for detecting anomalous behaviors based on process algebra. We specify the architecture of the behavior detection model and the detection algorithm. The anomalous behavior detection and analysis demonstrate that our method is a good discriminator between normal and anomalous behavior characteristics of distributed applications. Performance evaluation shows that the proposed method enhances efficiency without security degradation.

Introduction

Distributed applications, such as cloud services and banking systems, involve multiple nodes communicating via message passing and performing tasks simultaneously. While detecting anomalous behavior in single-node applications is relatively straightforward by monitoring system call sequences, distributed environments introduce significant challenges:

  1. Concurrency: Parallel events across different machines make it difficult to define a "normal" sequence.
  2. Complexity: The high volume of interactions complicates pinpointing compromised nodes.
  3. Non-determinism: Standard models often require performance-heavy backtracking when encountering multiple possible execution paths.

Methodology

The proposed framework transitions from raw binary code to a formal mathematical model through a multi-step process.

Overview of the anomalous behavior detection framework

Static Analysis and Process Algebra Transformation

The process begins with analyzing application binary code to generate Control Flow Graphs (CFGs). These graphs map every possible execution path. These CFGs are then converted into process algebra expressions. Process algebra provides a mathematical language to describe concurrent systems using operators for sequential action (.), choice (+), and parallel execution (||).

Control Flow Graphs (CFGs) for a distributed server-client system

Optimization and Runtime Monitoring

To ensure real-time detection speed, the system uses distributive laws to eliminate non-determinism in the algebraic expressions. This prevents the detection algorithm from needing to "guess" or backtrack. At runtime, the system monitors system calls and their arguments. By using "Process Migration Rules," the framework tracks the application's state across nodes. If a sequence of system calls deviates from the established algebraic model, an alarm is triggered.

Key Insights

  • Algebra as a "Gold Standard": Unlike learning-based systems that can be poisoned by noisy training data, this method uses the source code itself as the ground truth. If a behavior is not represented in the algebra, it is flagged as an anomaly.
  • Argument Stitching: The system analyzes system call arguments (e.g., socket descriptors) to link nodes. This allows it to "stitch" a send on one node to a recv on another, creating a holistic view of the distributed state.
  • Linear Complexity: By shifting computational complexity to the offline modeling phase, the runtime detection achieves linear space-time complexity, avoiding the performance bottlenecks of traditional IDS.

Evaluation and Results

The method was evaluated using a custom Bank Queue Management System and a Hadoop benchmark suite (WordCount, Sort, Pi).

  • Attack Precision: The system successfully detected Code Injection, Impossible Path, Man-in-the-Middle (MITM), and Remote Man-at-the-end (R-MATE) attacks, outperforming several existing Intrusion Detection Systems (IDS) in detecting MITM and R-MATE exploits.
  • Scalability and Efficiency: Performance tests on Hadoop showed that detection overhead grows linearly with the number of nodes. Runtime overhead remained low, generally under 10-15%, making it suitable for mission-critical systems.

Limitations and Future Work

A primary limitation is the susceptibility to Mimicry Attacks, where an attacker uses a "legal" sequence of system calls but with malicious data. Since the current model focuses on control flow logic rather than data flow content, it cannot detect these stealthy exploits. Future research will focus on integrating data flow analysis to address this gap.

Summary updated

Abstract

Distributed applications are composed of multiple nodes, which exchange information with individual nodes through message passing. Compared with traditional applications, distributed applications have more complex behavior patterns because a large number of interactions and concurrent behaviors exist among their distributed nodes. Thus, it is difficult to detect anomalous behaviors and determine the location and scope of abnormal nodes, and some attacks and misuse cannot be detected. To address this problem, we introduce a method for detecting anomalous behaviors based on process algebra. We specify the architecture of the behavior detection model and the detection algorithm. The anomalous behavior detection and analysis demonstrate that our method is a good discriminator between normal and anomalous behavior characteristics of distributed applications. Performance evaluation shows that the proposed method enhances efficiency without security degradation.
BibTeX
@inproceedings{ma2015anomalous,
  author = {Chuan Ma and Limin Shen and Tao Wang and Leslie Lamport and C Din and J Dovland and E Johnsen and O Olaf and M Gul and Hussain and C Collberg and S Martin and J Myers and J Nagra and F Idrees and M Rajarajan and A Memon and J Meseguer and P Lveczky and S Forrest and S Hofmeyr and A Somayaji and T Longstaff and S Hofmeyr and S Forrest and A Somayaji and P Helman and J Bhangoo and W Lee and S Stolfo and D Wagner and D Dean and J Giffin and S Jha and B Miller and R Gopalakrishna and E Spafford and J Vitek and F Jianming and T Fen and W Dan and H Feng and J Giffin and Y Huang and S Jha and M Moshirpour and A Mousavi and B Far and F Yang and T Aotain and H Masuhare and D Gupta and K Vishwanath and M Mcnett and J Tan and S Kavulya and R Gandhi and M Rohr and A Van Hoorn and W Hasselbring and M Moshirpour and R Alhajj and M Moussavi and B Far and G Milne and R Milner and J Baeten and R Milner and C Hoare and J Hopcroft},
  title = {{An Anomalous Behavior Detection Method Using System Call Sequences for Distributed Application}},
  booktitle = {KSII},
  year = {2015},
  doi = {10.3837/tiis.2015.02.010},
}
Figure