Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer

Chen Luo, Zhengzhang Chen, Lu-An Tang, Anshumali Shrivastava, Zhichun Li

Introduction

In modern enterprise security, dependency graphs serve as a critical foundation for forensic analysis and intrusion detection. These graphs map complex interactions between system entities—such as processes, files, and network sockets—to establish a baseline of "normal" behavior. However, a significant hurdle in deploying these systems is the "cold-start" problem. Building a mature, reliable dependency graph typically requires a "Learning Phase" of approximately 30 days of continuous monitoring. During this month-long window, the security system remains largely ineffective in new environments.

Traditional transfer learning techniques often fail in this domain because system event data is primarily categorical (e.g., process names, IP addresses) rather than numerical. Furthermore, every enterprise environment contains unique entities, making direct model migration impossible.

The ACRET Framework

To address these challenges, we introduce ACRET (Accelerating Dependency Graph Learning via Knowledge Transfer). ACRET enables the reuse of knowledge from a mature "source" environment to rapidly bootstrap a "target" environment, reducing the required learning time from weeks to just a few days.

Comparison between the traditional 30-day learning workflow and the accelerated 3-day ACRET workflow.

The framework operates through two primary components:

  1. Entity Estimation Model (EEM): This model handles the challenges of categorical data by embedding system entities into a latent vector space. By utilizing meta-paths (e.g., "Process-File-Process"), ACRET captures complex heterogeneous relationships. A key innovation is a hypothesis-testing mechanism that filters out irrelevant source entities, ensuring that only statistically significant knowledge is transferred to the new domain.
  2. Dependency Construction Model (DCM): This stage builds the target graph by solving a two-constraint optimization problem. It balances a Smoothness Constraint, which keeps the graph consistent with local target data, and a Consistency Constraint, which preserves the unique domain signature of the source to prevent simple cloning.

Experimental Evaluation

The effectiveness of ACRET was validated using both synthetic datasets and massive real-world enterprise logs, including a Windows dataset (120 million events) and a Linux dataset (10 million events).

Key Results

  • 10x Training Speedup: ACRET reduces the time needed for a dependency graph to become "deployable" from 30 days to just 3 days.
  • High Accuracy with Minimal Data: In intrusion detection case studies involving ten real-world attack types, ACRET achieved 60% precision and 73% recall after only three days of training. In contrast, models trained from scratch for the same duration yielded near-zero precision.
  • Efficiency: The underlying optimization algorithm is highly convergent, typically reaching a solution in fewer than 10 iterations, making it suitable for large-scale production environments.

Discussion and Future Work

ACRET represents a shift from "rebuilding" to "reusing" models in system security. By treating existing mature environments as informative priors, security teams can significantly decrease the window of vulnerability during new deployments.

Currently, ACRET assumes that the source and target domains share a similar underlying physical system (e.g., both are enterprise IT networks). Future research will explore cross-domain transfer (e.g., from IT to OT networks) and automated meta-path discovery to further enhance the framework's flexibility and depth of analysis.

Summary updated

Abstract

Dependency graph, as a heterogeneous graph representing the intrinsic relationships between different pairs of system entities, is essential to many data analysis applications, such as root cause diagnosis, intrusion detection, etc. Given a well-trained dependency graph from a source domain and an immature dependency graph from a target domain, how can we extract the entity and dependency knowledge from the source to enhance the target? One way is to directly apply a mature dependency graph learned from a source domain to the target domain. But due to the domain variety problem, directly using the source dependency graph often can not achieve good performance. Traditional transfer learning methods mainly focus on numerical data and are not applicable. In this paper, we propose ACRET, a knowledge transfer based model for accelerating dependency graph learning from heterogeneous categorical event streams. In particular, we first propose an entity estimation model to filter out irrelevant entities from the source domain based on entity embedding and manifold learning. Only the entities with statistically high correlations are transferred to the target domain. On the surviving entities, we propose a dependency construction model for constructing the unbiased dependency relationships by solving a two-constraint optimization problem. The experimental results on synthetic and real-world datasets demonstrate the effectiveness and efficiency of ACRET. We also apply ACRET to a real enterprise security system for intrusion detection. Our method is able to achieve superior detection performance at least 20 days lead lag time in advance with more than 70% accuracy.
BibTeX
@misc{luo2017accelerating,
  author = {Chen Luo and Zhengzhang Chen and Lu-An Tang and Anshumali Shrivastava and Zhichun Li},
  title = {{Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer}},
  year = {2017},
  eprint = {1708.07867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.07867},
}
Figure