Fuzzy Logic Model for Digital Forensics: A Trade-off between Accuracy, Complexity and Interpretability

Andrii ShalaginovIJCAI 2017 · DOI

TL;DR

This paper tackles the challenge of analyzing massive, complex datasets in digital forensics, where traditional methods struggle. They propose an improved Neuro-Fuzzy (NF) model that balances accuracy and interpretability, making findings more understandable in legal contexts while achieving higher performance.

What Problem Does This Paper Address?

Digital forensics investigations are increasingly hampered by the sheer volume, velocity, and complexity of data generated by emerging technologies. This "Big Data" problem makes it difficult for investigators to extract meaningful evidence and present it in a way that is comprehensible to legal professionals. Furthermore, many machine learning (ML) approaches produce models that are too complex to explain, failing to meet legal standards like the Daubert Criteria, which require scientific methods to be testable, have known error rates, be well-known, and accepted by the community. Existing Neuro-Fuzzy models, while promising for combining computational intelligence with human-understandable rules, often suffer from poor accuracy and overly complex models, making them unsuitable for forensic applications.

Methods

The paper focuses on improving a hybrid Neuro-Fuzzy (NF) architecture. NF combines Fuzzy Logic (FL) and Artificial Neural Networks (ANN) into a classification model. The process involves two main stages:

  1. 1st NF Stage (Unsupervised): This stage uses a Self-Organizing Map (SOM) to group similar data samples. The challenge here is determining the optimal size (width and height) of the SOM. The authors propose using exploratory data analysis to find optimal SOM parameters, moving away from the standard Vesanto method. They also introduce a new Gaussian Membership Function (MF) that is more robust to data variations compared to traditional triangular or Kosko MFs.
  2. 2nd NF Stage (Supervised): This stage converts the fuzzy patches (clusters from the SOM) into fuzzy rules, which are then used to train an ANN. The authors introduce a new defuzzification function and improved SOM node configuration to enhance performance, particularly for multinomial classification problems.

Datasets

  • KDD Cup 1999 dataset: A widely used benchmark for network intrusion detection.
  • SUSY and HIGGS datasets: Large-scale datasets used for evaluating classification performance.
  • Novel dataset: The authors created a new dataset consisting of 328,337 binaries categorized into 10,362 families and 35 categories of Windows PE32 viruses. This dataset was examined using both static and dynamic behavioral analysis.

Key Insights and Ideas

  • The Trade-off is Key: The core idea is that in digital forensics, there's a crucial trade-off between model accuracy and interpretability. The improved NF model aims to optimize this balance.
  • Exploratory Data Analysis for SOM: Instead of relying on fixed heuristics for SOM configuration, using exploratory data analysis allows for more tailored and effective clustering, leading to better fuzzy rule extraction.
  • Gaussian MFs for Robustness: The new Gaussian Membership Functions are more adaptable to the nuances of real-world data, improving the accuracy of fuzzy rule representation.
  • Addressing Multinomial Challenges: The paper specifically addresses the under-explored area of multinomial classification in digital forensics, showing improved performance with their revised NF.
  • Forensic Soundness: The goal is to create models that are not only accurate but also "forensically-sound," meaning they can be understood and validated in a legal setting, adhering to standards like Daubert.

Key Results

  • Reduced Complexity: The revised NF method significantly reduces the number of fuzzy rules. For the KDD Cup 1999 dataset, the original NF produced 15,081 rules, while the revised NF produced only 12 rules.
  • Improved Accuracy: The revised NF achieved higher accuracy on the KDD Cup 1999 dataset, reaching 98.9% compared to the original NF's 94.8%.
  • Reliability on Large Datasets: Experiments on KDD Cup 1999, SUSY, and HIGGS datasets demonstrated the reliability of the proposed improvements.
  • Handling Difficult Data: The revised NF was able to be trained on datasets where some other ML methods failed.

Limitations and Future Work

  • Limitations: The paper acknowledges that preliminary studies of the original NF showed poor accuracy and high complexity. While their improvements address this, the full extent of limitations for the revised model isn't deeply detailed beyond the performance gains.
  • Future Work: The authors express interest in exploring the synergy of Deep Learning and Neuro-Fuzzy for future research.

Summary updated

BibTeX
@inproceedings{shalaginov2017fuzzy,
  author = {Andrii Shalaginov},
  title = {{Fuzzy Logic Model for Digital Forensics: A Trade-off between Accuracy, Complexity and Interpretability}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/763},
}
Figure