This paper presents MODEXRE (Modular Digital Examination and Reporting Engine), a framework designed to move network intrusion detection from the "clean" environment of a research lab into the messy, high-stakes world of digital forensics. By combining explainable AI (XAI) with a cryptographically verifiable chain of custody, the authors demonstrate how to build a system that isn't just accurate, but also legally admissible.
TL;DR
MODEXRE is a forensic pipeline that bridges the gap between laboratory AI models and real-world evidence by using a dual-mode architecture (Laboratory vs. Formal) and a hash-chained digital chain of custody. The paper highlights how "perfect" benchmark models often fail in the field due to feature gaps and hidden bugs, and provides a roadmap for building AI systems that are transparent enough for the courtroom.
What Problem Does This Paper Address?
Most research in AI-based intrusion detection focuses on maximizing accuracy on curated datasets like CICIDS2017. However, these models often fail when deployed for actual forensic work. The authors identify three major hurdles:
- The Feature Gap: Laboratory datasets often have 70+ features, but real-world evidence (like firewall logs) might only provide 4 or 5.
- Admissibility: In a legal context, you must prove that the model used in a case is exactly the same one that was validated in the lab, and you must be able to explain why it flagged a specific event.
- Architectural Blindness: Standard event-by-event classifiers often miss attacks that only appear as patterns over time, such as port scanning.
Methods
The authors introduce a dual-mode architecture:
- Laboratory Mode: Where models are trained, tuned, and certified using synthetic and public data.
- Formal Mode: A "frozen" environment where a hash-verified model performs inference on real case evidence. No retraining is allowed here, ensuring the model's integrity.
Key Technical Components: * OCSF Normalization: Uses the Open Cybersecurity Schema Framework to translate heterogeneous logs (Suricata, CEF, PCAP) into a common format. * Causal Temporal Flow Aggregation: A layer that looks at windows of time (e.g., 60 seconds) to calculate features like "distinct destination ports contacted," allowing the model to "see" sequence-dependent attacks. * XGBoost + SHAP: Uses gradient boosting for classification and SHAP (SHapley Additive exPlanations) to provide per-prediction explanations, showing which features pushed the model toward a specific conclusion. * Digital Chain of Custody: Implements a Level 2 hash-chain (SHA-256) that links every step—from ingestion to the final report—ensuring the evidence hasn't been tampered with.
Datasets
The study utilizes three primary benchmark datasets for training and validation: CICIDS2017, UNSW-NB15, and Kitsune. To test the system's "Formal Mode," the authors used real-world evidence including Suricata alerts, CEF firewall logs, and raw packet captures (PCAP) generated from their own infrastructure.
Key Insights and Ideas
The most compelling parts of the paper are the "design decisions" triggered by empirical failures:
- The Timestamp Leak: The authors found that an initial model achieved a near-perfect 0.984 F1-score on the Kitsune dataset. SHAP analysis revealed the model wasn't learning traffic patterns; it was learning the absolute timestamps of the capture sessions—a forensically useless artifact.
- The "Honest" Performance Drop: When the model was restricted to the 5 features actually available in real logs (vs. the 78 in the lab), performance dropped, but the model became capable of correctly identifying real SSH brute-force attacks that the "full" model missed.
- The Silent Binary Bug: During testing against a real packet capture, the authors discovered a bug where the pipeline was silently training a binary classifier (Malicious vs. Normal) even when multiclass labels were available. This was only caught by confronting the model with real-world data, as it was invisible in standard benchmark tests.
Key Results
- Temporal Aggregation: Adding the temporal aggregation layer caused the multiclass F1-macro on the Kitsune dataset to jump from 0.394 to 0.932, proving that context is king for certain attack types.
- Model Performance: The default certified model (v1) achieved an F1-macro of 0.9538 on 4 classes. A more complex 18-category model (v3) reached 0.6214, a figure the authors argue is a more "honest" reflection of the difficult multiclass task in a forensic setting.
- Port-Scan Detection: Specialized binary detectors for port scanning reached an impressive 0.9997 F1-macro.
Limitations and Future Work
- Chain of Custody: The current system uses Level 2 (hash-chaining). The authors aim to move to Level 3 by incorporating digital signatures for non-repudiation.
- Legal Scrutiny: While designed for regulatory alignment (like the EU AI Act), MODEXRE has not yet been evaluated by legal practitioners in an actual judicial process.
- Attack Diversity: The empirical validation focused heavily on port scanning and reconnaissance; other categories like shellcode or backdoors require further depth of analysis.
Summary updated