Towards Automatic Hands-on-Keyboard Attack Detection Using LLMs in EDR Solutions

Amit Portnoy, Ehud Azikri, Shay Kels

Abstract

Detecting "Hands-on-Keyboard" (HOK) cyberattacks remains a significant challenge for Endpoint Detection and Response (EDR) systems due to the noise of raw telemetry and the ability of human adversaries to blend in with legitimate administrative activity. This paper presents a novel framework that transforms unstructured endpoint logs into human-readable "endpoint stories" and utilizes a pipeline of Large Language Models (LLMs), specifically Phi-2 and BERT, for classification. Our ensemble approach achieves a 75.4% True Positive Rate at a 1% False Positive Rate, significantly outperforming traditional machine learning baselines and demonstrating the efficacy of LLMs in inferring attacker intent from narrative sequences.

Introduction

In the modern cybersecurity landscape, Hands-on-Keyboard (HOK) attacks represent a critical threat. Unlike automated malware, HOK attacks involve human adversaries who actively maneuver through a network, often using legitimate system tools—a technique known as "living off the land." These attackers are meticulous, making them nearly invisible to traditional signature-based detection or simple anomaly sensors.

The primary obstacle to detecting these attacks is the nature of the data. EDR systems generate massive volumes of raw telemetry, including process starts, file deletions, and registry changes. This data is often unstructured and lacks the context necessary for standard machine learning models to distinguish between a busy system administrator and a malicious intruder.

Methodology: From Raw Logs to Endpoint Stories

The proposed framework consists of two main stages: Narrative Transformation and LLM-based Classification.

Narrative Transformation

Instead of feeding raw logs directly into a model, the researchers transform telemetry into structured narratives called "endpoint stories." This process involves:

  • Aggregation & Filtering: Grouping events by timeframe and removing noise that does not contribute to HOK detection.
  • Rephrasing: Converting technical logs into a consistent, natural language format (e.g., "user X started process Y").
  • Deduplication & Normalization: Grouping repetitive events and replacing long file paths or GUIDs with generic placeholders to maintain a concise narrative.
Example of transformed endpoint stories used as input for the LLM pipeline.

The Model Pipeline

To handle the long-context nature of these stories, the authors employed a windowing approach with two primary architectures:

  1. BERT + LSTM: A small BERT model creates embeddings for windows of the story, which are then processed by a bidirectional LSTM to capture the sequence of events.
  2. Phi-2 + Transformer: A fine-tuned Phi-2 (2.7B parameter) model generates window embeddings, which are then fed into a 4-layer Transformer head for final classification.

The researchers found that an ensemble averaging the scores of both the BERT and Phi-2 approaches yielded the best performance.

Key Insights

  • Context and Intent: LLMs excel in this domain because they can infer intent from a narrative. By treating logs as stories, the models utilize pre-trained reasoning capabilities to identify suspicious sequences that keyword-based systems might overlook.
  • Managing Long Contexts: By breaking stories into windows and using secondary classification heads (LSTM or Transformer), the framework effectively analyzes long-running attacks without requiring excessive compute resources.
  • Efficiency of Small Models: The study demonstrates that smaller, specialized models like Phi-2 and BERT-small are highly effective and more feasible for the low-latency requirements of production EDR systems than massive models like GPT-4.

Experimental Results

The models were evaluated using real-world data from global EDR deployments, focusing on the True Positive Rate (TPR) at a 1% False Positive Rate (FPR). Maintaining a low FPR is essential in EDR to prevent alert fatigue among security analysts.

Performance comparison of various models on the HOK detection task, highlighting the superiority of the LLM ensemble.

As shown in the results, the LLM-based ensemble significantly outperformed the traditional LightGBM baseline, catching approximately 23% more attacks at the same strict 1% false-positive threshold.

Conclusion and Future Work

While the results are promising, several challenges remain for real-world deployment. The computational weight of LLMs compared to traditional models presents latency hurdles in high-volume environments. Additionally, improving the interpretability of model decisions and ensuring robustness against adversarial evolution—where attackers craft narratives to appear benign—are key areas for future research. Nonetheless, this work marks a significant step toward automated, intent-aware threat detection in enterprise environments.

Summary updated

Abstract

Endpoint Detection and Remediation (EDR) platforms are essential for identifying and responding to cyber threats. This study presents a novel approach using Large Language Models (LLMs) to detect Hands-on-Keyboard (HOK) cyberattacks. Our method involves converting endpoint activity data into narrative forms that LLMs can analyze to distinguish between normal operations and potential HOK attacks. We address the challenges of interpreting endpoint data by segmenting narratives into windows and employing a dual training strategy. The results demonstrate that LLM-based models have the potential to outperform traditional machine learning methods, offering a promising direction for enhancing EDR capabilities and apply LLMs in cybersecurity.
BibTeX
@misc{portnoy2024towards,
  author = {Amit Portnoy and Ehud Azikri and Shay Kels},
  title = {{Towards Automatic Hands-on-Keyboard Attack Detection Using LLMs in EDR Solutions}},
  year = {2024},
  eprint = {2408.01993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.01993},
}
Figure