Abstract
Detecting "Hands-on-Keyboard" (HOK) cyberattacks remains a significant challenge for Endpoint Detection and Response (EDR) systems due to the noise of raw telemetry and the ability of human adversaries to blend in with legitimate administrative activity. This paper presents a novel framework that transforms unstructured endpoint logs into human-readable "endpoint stories" and utilizes a pipeline of Large Language Models (LLMs), specifically Phi-2 and BERT, for classification. Our ensemble approach achieves a 75.4% True Positive Rate at a 1% False Positive Rate, significantly outperforming traditional machine learning baselines and demonstrating the efficacy of LLMs in inferring attacker intent from narrative sequences.
Introduction
In the modern cybersecurity landscape, Hands-on-Keyboard (HOK) attacks represent a critical threat. Unlike automated malware, HOK attacks involve human adversaries who actively maneuver through a network, often using legitimate system tools—a technique known as "living off the land." These attackers are meticulous, making them nearly invisible to traditional signature-based detection or simple anomaly sensors.
The primary obstacle to detecting these attacks is the nature of the data. EDR systems generate massive volumes of raw telemetry, including process starts, file deletions, and registry changes. This data is often unstructured and lacks the context necessary for standard machine learning models to distinguish between a busy system administrator and a malicious intruder.
Methodology: From Raw Logs to Endpoint Stories
The proposed framework consists of two main stages: Narrative Transformation and LLM-based Classification.
Narrative Transformation
Instead of feeding raw logs directly into a model, the researchers transform telemetry into structured narratives called "endpoint stories." This process involves:
- Aggregation & Filtering: Grouping events by timeframe and removing noise that does not contribute to HOK detection.
- Rephrasing: Converting technical logs into a consistent, natural language format (e.g., "user X started process Y").
- Deduplication & Normalization: Grouping repetitive events and replacing long file paths or GUIDs with generic placeholders to maintain a concise narrative.
The Model Pipeline
To handle the long-context nature of these stories, the authors employed a windowing approach with two primary architectures:
- BERT + LSTM: A small BERT model creates embeddings for windows of the story, which are then processed by a bidirectional LSTM to capture the sequence of events.
- Phi-2 + Transformer: A fine-tuned Phi-2 (2.7B parameter) model generates window embeddings, which are then fed into a 4-layer Transformer head for final classification.
The researchers found that an ensemble averaging the scores of both the BERT and Phi-2 approaches yielded the best performance.
Key Insights
- Context and Intent: LLMs excel in this domain because they can infer intent from a narrative. By treating logs as stories, the models utilize pre-trained reasoning capabilities to identify suspicious sequences that keyword-based systems might overlook.
- Managing Long Contexts: By breaking stories into windows and using secondary classification heads (LSTM or Transformer), the framework effectively analyzes long-running attacks without requiring excessive compute resources.
- Efficiency of Small Models: The study demonstrates that smaller, specialized models like Phi-2 and BERT-small are highly effective and more feasible for the low-latency requirements of production EDR systems than massive models like GPT-4.
Experimental Results
The models were evaluated using real-world data from global EDR deployments, focusing on the True Positive Rate (TPR) at a 1% False Positive Rate (FPR). Maintaining a low FPR is essential in EDR to prevent alert fatigue among security analysts.
As shown in the results, the LLM-based ensemble significantly outperformed the traditional LightGBM baseline, catching approximately 23% more attacks at the same strict 1% false-positive threshold.
Conclusion and Future Work
While the results are promising, several challenges remain for real-world deployment. The computational weight of LLMs compared to traditional models presents latency hurdles in high-volume environments. Additionally, improving the interpretability of model decisions and ensuring robustness against adversarial evolution—where attackers craft narratives to appear benign—are key areas for future research. Nonetheless, this work marks a significant step toward automated, intent-aware threat detection in enterprise environments.
Summary updated