BAGUETTE: Hunting for Evidence of Malicious Behavior in Dynamic Analysis Reports

Vincent Raulin, Pierre-François Gimenez, Yufei Han, Valérie Viet Triem TongSECRYPT 2023 · DOI

TL;DR

This paper introduces BAGUETTE, a novel graph-based representation for dynamic malware analysis reports. By transforming complex execution traces into a structured, heterogeneous graph, BAGUETTE enables both automated detection of suspicious behaviors using "metagraphs" and more intuitive manual analysis for security experts.

What Problem Does This Paper Address?

Malware analysis is crucial for cybersecurity, but current dynamic analysis methods often produce voluminous, hard-to-digest reports. These reports, while rich in information, make it difficult for both human analysts and automated systems to quickly identify malicious activities. Existing approaches struggle to provide a semantic understanding of malware behavior, making them vulnerable to evasion techniques. The challenge is to create a representation that is both comprehensive and interpretable, bridging the gap between raw execution data and actionable insights for malware detection and analysis.

Methods

The core of BAGUETTE is a heterogeneous graph that models the interactions of a malware sample with its host system during dynamic analysis. This graph comprises 14 types of nodes, including Host, Process, Thread, API Call, File, Network Connection, and Registry Keys, along with various edge types to represent relationships between them.

The construction of a BAGUETTE graph involves three steps:

  1. Initialization: Creating nodes for Host, Process, Thread, and API calls, linking them according to their execution hierarchy.
  2. Chronological Processing: Iterating through API calls and creating nodes for File, Handle, Socket, Connection, Registry Key, and Import, connecting them to the relevant API calls. Special cases like injected threads and direct file path references are handled.
  3. Refinement: Building file system trees with Directory nodes, creating "Diff" nodes to track data flow and similarity, and adding "References File System" edges for registry entries pointing to files.

To facilitate automated analysis, the paper introduces metagraphs, which are subgraph patterns representing specific suspicious behaviors (e.g., "High-Entropy Writing," "Covert Execution"). The system then searches for matches of these metagraphs within the BAGUETTE graphs.

For visualization, a graph pruning technique is employed to reduce the complexity of large graphs, focusing on key interactions and removing less informative elements.

Datasets

The authors used a dataset of Cuckoo reports from malware samples of three families collected from MalwareBazaar:

  • GCleaner: 247 samples (file dropper)
  • SnakeKeyLogger: 436 samples (key logger and spyware)
  • LockBit: 7 samples (ransomware)

The BAGUETTE graphs generated from these samples are made available in their repository.

Key Insights and Ideas

  • Bridging the Gap: BAGUETTE effectively translates raw, sequential execution traces into a structured, interconnected graph, making complex interactions more understandable.
  • Metagraphs as Behavioral Signatures: The concept of metagraphs provides a powerful way to define and automatically detect specific malicious behaviors, acting as a form of behavioral signature.
  • Granularity Matters: The heterogeneous nature of BAGUETTE, with its numerous node and edge types, allows for a much more detailed and semantically rich representation of malware behavior compared to simpler sequential or object-based approaches.
  • Visualization Aids Analysis: Pruning and highlighting techniques in graph visualization are essential for making large, complex malware analysis graphs manageable and interpretable for human experts.
  • Evasion Detection Challenges: The study highlights that current metagraphs are not designed to capture all evasion techniques, suggesting a need for more sophisticated patterns in the future.

Key Results

  • All five proposed metagraph patterns could be matched in at least one BAGUETTE graph from the tested malware families, indicating their potential as behavioral indicators.
  • The distribution of metagraph matches varied significantly across malware families, suggesting that metagraphs can serve as behavioral signatures for malware families.
  • For GCleaner, metagraph patterns (except "Auto-Run") showed high match rates and low variance, indicating characteristic behaviors.
  • LockBit samples showed a very high number of "High-Entropy Writing" matches (average 2450), strongly indicating ransomware encryption payloads.
  • SnakeKeyLogger samples often produced "almost empty" BAGUETTE graphs, suggesting effective evasion techniques that current metagraphs could not capture.

Limitations and Future Work

  • Evasion Techniques: The current metagraphs are not designed to capture all evasion behaviors, as seen with the SnakeKeyLogger samples. Future work should focus on enriching metagraph patterns to identify these evasive techniques.
  • Metagraph Complexity vs. Flexibility: There's a trade-off between the accuracy and flexibility of metagraphs. Simpler metagraphs are more flexible but might lead to more false positives, while complex ones are more accurate but harder to match.
  • Algorithm for Learning Metagraphs: The authors suggest developing an algorithm to learn significant metagraphs automatically to better separate malware datasets.
  • Clustering and Classification: Future work will involve using metagraphs to identify clusters of malware with similar behaviors and to qualify unknown malware by recognizing partially known behaviors.

Summary updated

BibTeX
@inproceedings{raulin2023baguette,
  author = {Vincent Raulin and Pierre-François Gimenez and Yufei Han and Valérie Viet Triem Tong},
  title = {{BAGUETTE: Hunting for Evidence of Malicious Behavior in Dynamic Analysis Reports}},
  booktitle = {SECRYPT},
  year = {2023},
  doi = {10.5220/0012086400003555},
}
Figure