Prioritization and Alert Fusion in Distributed IoT Sensors Using Kademlia Based Distributed Hash Tables

Mansoor Nasir, Khan Muhammad, Paolo Bellavista, Mi Young Lee, Muhammad SajjadIEEE Access 2020 · OpenReview

Abstract

Distributed intrusion detection systems (IDS) are essential for monitoring and responding to anomalies in real-time across large networks. However, the proliferation of IoT devices often leads to an overwhelming volume of alerts and a high rate of false alarms. This paper proposes a framework to mitigate "alert fatigue" by limiting message generation without sacrificing detection accuracy. The system utilizes fog-level nodes for alert collection and analysis, employs a multi-factor prioritization scheme, and leverages Kademlia-based Distributed Hash Tables (DHT) for efficient alert fusion. Experimental results using the DARPA 1999 dataset demonstrate a 62% reduction in message traffic and a false positive accuracy exceeding 80%.

Introduction

As the Internet of Things (IoT) continues to expand, security becomes a paramount concern. Distributed Intrusion Detection Systems (DIDS) are deployed to safeguard these networks, yet they frequently suffer from two critical issues: alert flooding and high false-positive rates. Coordinated attacks can trigger thousands of redundant alerts, overwhelming both network bandwidth and security analysts. Centralized processing of these alerts often creates bottlenecks and single points of failure. This research addresses these challenges by introducing a decentralized framework that processes and fuses alerts at the edge of the network (the "Fog" layer) before they reach the core.

Proposed Framework

The architecture consists of three primary layers: IoT sensors, Fog nodes, and mid-level "Collectors." The framework focuses on reducing redundancy and improving the quality of reported security events through a structured pipeline involving normalization, prioritization, filtering, and fusion.

The proposed framework architecture, illustrating the flow from IoT sensors through normalization and Kademlia-based fusion to final reporting.

Multi-Factor Prioritization

To distinguish critical threats from noise, the system calculates a "Cumulative Value" (cv) for each alert based on five key factors:

  1. Target Importance: The criticality of the asset under attack.
  2. Event Severity: Standardized scores (e.g., CVE) for the detected vulnerability.
  3. Detection Confidence: The reliability of the specific sensor's detection algorithm.
  4. Alert Correlation: Whether the alert fits into a known multi-step attack pattern.
  5. Service History: The frequency of recent attacks against the targeted service.

Alerts that do not cross a specific threshold are filtered out, while high-priority alerts proceed to the fusion stage. A rescoring mechanism provides feedback to refine the normalization process based on correlation results.

Kademlia-based Alert Fusion

A core innovation is the use of Kademlia-based Distributed Hash Tables (DHTs) for alert routing. By using attack characteristics (such as the destination port) as the DHT key, the network inherently routes related alerts to the same collector node. This decentralized approach ensures that similar alerts are fused into a single, comprehensive report, significantly reducing the volume of data sent to the core network. Kademlia was specifically chosen for its resilience to node churn and lower message overhead compared to other DHT protocols like Chord.

Evaluation and Results

The framework was validated using the DARPA 1999 dataset and real-world DShield logs. The DShield logs, comprising over 600 million entries, helped confirm the stability of port scan patterns and the efficiency of the DHT-based load balancing.

Key findings include:

  • Traffic Reduction: The system achieved a 62% reduction in the total number of generated messages.
  • Detection Accuracy: False positive accuracy was maintained at over 80%.
  • Priority Granularity: Unlike standard tools like Snort, which often provide broad priority categories, the proposed system successfully mapped alerts to a 1-to-10 scale, enabling more precise resource allocation.
  • Efficiency: Mathematical modeling confirmed that Kademlia's routing overhead ($n + \log_2 N$) is significantly lower than Chord’s ($n \cdot \log_2 N$) in high-traffic IoT scenarios.

Conclusion

By shifting alert processing and fusion to the fog layer, the proposed framework effectively addresses the scalability issues of modern DIDS. The combination of context-aware prioritization and DHT-based routing provides a robust solution for managing IoT security traffic. Future work will focus on optimizing the computational costs of prioritization, integrating modern IoT-specific datasets, and transitioning to non-structured storage solutions like MongoDB to better handle high-velocity data.

Summary updated

Abstract

Distributed intrusion detection systems (IDS) are primarily deployed across the network to monitor, detect, and report anomalies, as well as to respond in real-time. Predominantly, an IDS is equipped with a set of rules that it needs to infer to be able to perform efficient detection. However, reducing the generation of false alarms is a major challenge in any IDS implementation. Additionally, the sheer number of IoT devices that generate alarms in a moderately large sensor network may be overwhelming. In order to reduce alarms, this paper contributes to the field by proposing an original framework that limits the number of generated messages without compromising detection accuracy. The primary idea is to exploit mid-level nodes called collectors where similar alerts are collected and analyzed independently. Priority is assigned to each alert and similar alerts are fused to respective collectors for more informed decision making. In addition, Kademlia based Distributed Hash Table (DHT) is used for efficient alert transportation and distributed fusion of similar alerts. In order to minimize false alarm rate, event correlation is used to find similarity between events fused by different detection sensors. The framework is implemented in a fog-based environment to assess and evaluate the efficiency of the proposed system in edge network. The architecture is evaluated with the recognized DARPA 1999 dataset; the reported results show that the proposed technique reduces message generation by 62% while achieving false positive accuracy over 80%.
BibTeX
@inproceedings{nasir2020prioritization,
  author = {Mansoor Nasir and Khan Muhammad and Paolo Bellavista and Mi Young Lee and Muhammad Sajjad},
  title = {{Prioritization and Alert Fusion in Distributed IoT Sensors Using Kademlia Based Distributed Hash Tables}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=rv9l8vQxbz},
}
Figure