Abstract
Network attacks are increasingly prevalent, making it difficult to accurately detect intrusion activities using traditional approaches. This paper presents an integrated system for website security analysis based on the Hadoop framework. The core hardware components include an Intrusion Detection System (IDS) and a Hadoop cluster, facilitating data interaction between the IDS controller, detection engine, web server, and storage nodes. By revising the MapReduce Word Count algorithm, the system extracts and analyzes statistical data such as IP addresses, intrusion levels, types, and timestamps. These results are stored in a MySQL database and presented through a visualization layer to assist website managers in monitoring security status effectively.
Introduction
As network attacks grow in sophistication, traditional intrusion detection systems (IDS) often struggle with the sheer volume and complexity of generated log data. Many organizations lack the specialized expertise required to interpret raw security logs, and standalone detection methods frequently fail to provide a holistic view of a website's security posture.
A critical gap exists between theoretical big data research and practical implementation. While technologies like Hadoop are frequently used in simulations with static datasets, there is a pressing need for integrated systems capable of handling real-world network operations. This research aims to bridge that gap by developing a system that cleans, analyzes, and visualizes "big data" from security hardware, transforming raw logs into actionable intelligence for human administrators.
System Architecture
The proposed system utilizes a multi-layered architecture divided into hardware and software subsystems to ensure scalability and efficient data processing.
Hardware Subsystem
The hardware infrastructure consists of a website server, an IDS engine and controller, and a four-node Hadoop cluster running CentOS 6.5 and Hadoop 2.6. One node is dedicated to a MySQL database for storing processed results. The network topology ensures seamless data flow between the detection engines and the distributed computing cluster.
Software Subsystem
The software architecture is organized into three primary layers: the Data Storage layer, the Business Computing layer, and the Application Service layer. This modular approach allows for efficient data transmission and parallel computing.
Methodology and Data Processing
The system implements a robust data pipeline to handle live network logs:
- Acquisition: Logs are captured from the IDS engine via FTP and uploaded to the Hadoop Distributed File System (HDFS).
- Cleaning and Parsing: A MapReduce program processes the logs line-by-line, extracting critical features: source IP address, entry time, attack type, and danger level.
- Statistical Analysis: The authors adapted the standard MapReduce Word Count algorithm. Instead of counting linguistic tokens, the system performs frequency analysis on the extracted security features to identify attack patterns.
- Presentation: Results are pushed to MySQL and rendered via a J2EE-based web interface using the Echarts library for interactive visualization.
Unlike many studies relying solely on benchmarks like KDD-CUP99, this implementation was validated using real-world log data from a university website, ensuring the system can handle the noise and complexity of live traffic.
Key Insights and Results
The integration of Hadoop and visualization tools yielded several significant findings:
- Heuristic Adaptation: Repurposing the Word Count model allowed the system to leverage Hadoop’s native efficiency for frequency analysis without requiring a bespoke computational paradigm.
- Temporal Vulnerabilities: Analysis revealed that intrusion attempts are not constant. Data visualization indicated that danger levels peak during specific windows, notably around midnight on Wednesdays and Saturdays, suggesting automated botnet activity.
- Operational Efficiency: The distributed cluster demonstrated the ability to process large-scale datasets that would overwhelm traditional relational databases, providing a multi-dimensional view of security threats (IP, frequency, and severity).
Conclusion
This research demonstrates the feasibility of using Hadoop-based frameworks to enhance website security through automated log analysis and visualization. By bridging the "expertise gap," the system allows managers to understand complex security postures at a glance. Future work will explore more sophisticated algorithms using the Mahout library for predictive threat detection and investigate streaming technologies like Apache Spark to reduce latency for real-time attack response.
Summary updated