Hadoop-based Intrusion Detection Technology and Data Visualization for Website Security

Xiaoming Zhang, Yuxin Wang, Ge-tong Zhang, Guang Wang2017 · DOI

Abstract

Network attacks are increasingly prevalent, making it difficult to accurately detect intrusion activities using traditional approaches. This paper presents an integrated system for website security analysis based on the Hadoop framework. The core hardware components include an Intrusion Detection System (IDS) and a Hadoop cluster, facilitating data interaction between the IDS controller, detection engine, web server, and storage nodes. By revising the MapReduce Word Count algorithm, the system extracts and analyzes statistical data such as IP addresses, intrusion levels, types, and timestamps. These results are stored in a MySQL database and presented through a visualization layer to assist website managers in monitoring security status effectively.

Introduction

As network attacks grow in sophistication, traditional intrusion detection systems (IDS) often struggle with the sheer volume and complexity of generated log data. Many organizations lack the specialized expertise required to interpret raw security logs, and standalone detection methods frequently fail to provide a holistic view of a website's security posture.

A critical gap exists between theoretical big data research and practical implementation. While technologies like Hadoop are frequently used in simulations with static datasets, there is a pressing need for integrated systems capable of handling real-world network operations. This research aims to bridge that gap by developing a system that cleans, analyzes, and visualizes "big data" from security hardware, transforming raw logs into actionable intelligence for human administrators.

System Architecture

The proposed system utilizes a multi-layered architecture divided into hardware and software subsystems to ensure scalability and efficient data processing.

Hardware Subsystem

The hardware infrastructure consists of a website server, an IDS engine and controller, and a four-node Hadoop cluster running CentOS 6.5 and Hadoop 2.6. One node is dedicated to a MySQL database for storing processed results. The network topology ensures seamless data flow between the detection engines and the distributed computing cluster.

Network topology of the integrated IDS and Hadoop system

Software Subsystem

The software architecture is organized into three primary layers: the Data Storage layer, the Business Computing layer, and the Application Service layer. This modular approach allows for efficient data transmission and parallel computing.

Layered software architecture for data storage, computing, and service

Methodology and Data Processing

The system implements a robust data pipeline to handle live network logs:

  1. Acquisition: Logs are captured from the IDS engine via FTP and uploaded to the Hadoop Distributed File System (HDFS).
  2. Cleaning and Parsing: A MapReduce program processes the logs line-by-line, extracting critical features: source IP address, entry time, attack type, and danger level.
  3. Statistical Analysis: The authors adapted the standard MapReduce Word Count algorithm. Instead of counting linguistic tokens, the system performs frequency analysis on the extracted security features to identify attack patterns.
  4. Presentation: Results are pushed to MySQL and rendered via a J2EE-based web interface using the Echarts library for interactive visualization.

Unlike many studies relying solely on benchmarks like KDD-CUP99, this implementation was validated using real-world log data from a university website, ensuring the system can handle the noise and complexity of live traffic.

Key Insights and Results

The integration of Hadoop and visualization tools yielded several significant findings:

  • Heuristic Adaptation: Repurposing the Word Count model allowed the system to leverage Hadoop’s native efficiency for frequency analysis without requiring a bespoke computational paradigm.
  • Temporal Vulnerabilities: Analysis revealed that intrusion attempts are not constant. Data visualization indicated that danger levels peak during specific windows, notably around midnight on Wednesdays and Saturdays, suggesting automated botnet activity.
  • Operational Efficiency: The distributed cluster demonstrated the ability to process large-scale datasets that would overwhelm traditional relational databases, providing a multi-dimensional view of security threats (IP, frequency, and severity).

Conclusion

This research demonstrates the feasibility of using Hadoop-based frameworks to enhance website security through automated log analysis and visualization. By bridging the "expertise gap," the system allows managers to understand complex security postures at a glance. Future work will explore more sophisticated algorithms using the Mahout library for predictive threat detection and investigate streaming technologies like Apache Spark to reduce latency for real-time attack response.

Summary updated

Abstract

Network attack becomes more popular today. It is still difficult to detect the intrusion activity accurately for users websites through traditional approaches. A kind of integrated system is designed and established for the website security analysis based on Hadoop system. The core components of hardware subsystem is intrusion detection system (IDS) and Hadoop cluster. The data can be interacted between the intrusion detection system (IDS) controller, detection engine, website server, transferring server and Hadoop cluster. Based on the Map Reduce model, the Word Count algorithm is revised to obtain data statistical results such as IP address, intrusion level, intrusion type and intrusion time. These analyzed results are saved automatically into the MySQL database to form data tables, including statistical analysis, level distribution, week analysis and danger efficiency. Then, the resulted data are presented with visualization effect for the website managers.
BibTeX
@misc{zhang2017hadoop,
  author = {Xiaoming Zhang and Yuxin Wang and Ge-tong ZHANG and Guang Wang},
  title = {{Hadoop-based Intrusion Detection Technology and Data Visualization for Website Security}},
  year = {2017},
  doi = {10.2991/cnct-16.2017.12},
}
Figure