TL;DR
Lev-3B is a compact, open-source 3B-parameter model specifically optimized for log analysis tasks like anomaly detection and security classification. By employing a two-stage training curriculum—moving from general natural language decisions to a diverse mixture of log-specific tasks—it outperforms massive frontier models on anomaly detection at a significantly lower serving cost.
What Problem Does This Paper Address?
Automated log analysis is a cornerstone of system reliability and security, yet it remains notoriously difficult for machine learning models. The challenges are three-fold:
- Heterogeneity: Every logging system has its own vocabulary, templates, and field conventions. A model trained on one system often fails when moved to another.
- Class Imbalance: Anomalies are rare. A model that is "mostly right" can still overwhelm human operators with false positives or, worse, confidently miss a critical failure.
- Operational Constraints: Logs are high-volume and sensitive. Prompting "frontier" models (like GPT-4) via API is often too expensive, slow, and poses privacy risks, while these models typically return text labels rather than the calibrated probabilities needed for alert triage.
The authors introduce Lev-3B to bridge this gap, focusing on a "data recipe" that allows a small model to generalize across different logging environments.
Methods
Lev-3B is built on the Granite-4.1-3B-Base backbone and utilizes a shared pointer readout (based on the Kev architecture). Instead of generating free-form text, the model calculates a probability distribution over a fixed set of provided options (e.g., "Is this session normal or anomalous?").
The core contribution is the two-stage training recipe:
- Stage 1: Natural Language Decision Adaptation: The model is trained on a suite of standard NLP tasks (sentiment analysis, topic classification, etc.) to learn the mechanics of choosing between bounded options. Crucially, no log data is used here.
- Stage 2: Joint Log Adaptation: The model is trained on a mixture of three data types:
- Anomaly Data (A): Specific labels for system failures.
- ATT&CK Data (T): Security technique classification, augmented with synthetic examples to handle rare attack types.
- General Log Data (L): 35 auxiliary tasks (like template classification or summarization) across various log formats. This "generalist" exposure is designed to help the model handle logging systems it hasn't seen before.
Datasets
The paper utilizes a wide array of datasets for both training and rigorous "source-excluded" evaluation:
- Represented Sources (Seen during training): BGL, Spirit, and Thunderbird (HPC logs); specific ATT&CK sourcetypes from Splunk’s attack-data repository.
- Excluded Sources (Unseen during training/selection): HDFS (Hadoop) logs and 18 withheld ATT&CK sourcetypes. These are used to test true cross-source generalization.
- Auxiliary Data: The Decision-v7 suite (for Stage 1) and LogGLUE/LogQA (for the general log data in Stage 2).
Key Insights and Ideas
- Adaptation Beats Prompting: For the specialized domain of logs, a 3B model adapted via the authors' recipe far outperforms zero-shot or few-shot prompting of much larger frontier models.
- The Power of "General" Logs: A key finding is that adding general log data (Stage 2, component L)—even if unrelated to the specific target task—dramatically improves the model's ability to transfer its skills to completely new logging systems.
- Task Interference: The authors noted that adding security classification (ATT&CK) initially degraded anomaly detection performance. However, the inclusion of general log data recovered most of these losses, suggesting it helps the model manage its internal capacity more effectively.
- The Confidence Trap: While the model is well-calibrated on familiar data, it exhibits a specific failure mode on unseen systems: it "confidently misses" anomalies, labeling them as normal with high probability. This highlights that calibration on one system does not guarantee reliability on another.
Key Results
- Anomaly Detection: Lev-3B achieved an F1 score of 88–99 on represented sources. On the unseen HDFS source, it reached an F1 of 43.3, nearly double the performance of the best-prompted frontier model (GPT-5.6) and four times the AUPRC of hosted decision models.
- Security Classification: It outperformed all hosted decision models (Jev, OpenAI Decisions) but trailed behind prompted GPT-5.6, likely because the frontier models possess broader prior knowledge of security techniques.
- Efficiency: When served on an NVIDIA H100, Lev-3B's latency for anomaly detection dropped to 172 ms, making it cost-competitive with or cheaper than API-based solutions for sustained workloads.
Limitations and Future Work
- Generalization Limits: While Lev-3B handles unseen sources better than its peers, the drop in performance from "seen" to "unseen" systems remains significant.
- Confident Errors: The model's tendency to confidently miss anomalies on new systems is a major hurdle for production deployment. The authors suggest that future work should focus on calibration techniques that can adapt to a target source without requiring labels.
- Data Dependency: The model’s security classification skill is heavily tied to the volume of real training examples; synthetic data helped but did not fully close the gap for rare techniques.
The authors have released the checkpoint, recipe, and data manifests to encourage reproducible research in compact, specialized models for system observability.
Summary updated