Enhancing web traffic attacks identification through ensemble methods and feature selection

Daniel Urda, Branly Martínez, Nuño Basurto, Meelis Kull, Ángel Arroyo, Álvaro HerreroarXiv 2024 · DOI · arXiv

Abstract

This paper evaluates the effectiveness of ensemble machine learning methods, specifically Random Forest and XGBoost, in identifying malicious web traffic. By transforming raw HTTP traces into a rich set of 78 features and applying feature selection techniques, the study demonstrates that ensemble models significantly outperform traditional baseline classifiers. Results show that XGBoost achieves a near-perfect Area Under the ROC Curve (AUC) of 0.989, offering a robust framework for securing e-commerce platforms against common web-based attacks.

Introduction

Websites serve as the primary interface for e-commerce but are also the frontline of cyber warfare, facing constant threats from SQL injections, cross-site scripting (XSS), and DDoS attacks. While large enterprises can deploy sophisticated security suites, small- and medium-sized enterprises (SMEs) often struggle with the cost and technical complexity of maintaining secure systems.

Existing research into Machine Learning (ML) for intrusion detection frequently relies on outdated feature sets or "baseline" algorithms like Support Vector Machines (SVM) that may struggle with the complexity and imbalance of real-world web traffic. This paper addresses this gap by proposing a methodology that extracts richer information from raw HTTP requests and leverages ensemble learning to create more reliable detection models.

Methodology

The researchers developed a multi-stage pipeline focused on feature engineering and ensemble classification.

Feature Extraction and Selection

Instead of analyzing raw logs directly, the team preprocessed HTTP traces by grouping samples into sessions. String-based data, such as URLs and payloads, were transformed into 78 distinct numerical and categorical features. This process included one-hot encoding for HTTP methods and calculating string lengths for specific key-value pairs within the payload.

Example of payload feature extraction, showing the transformation from raw string to processed numerical features.

To optimize the models, three feature selection (FS) methods were tested: Information Gain (IG), LASSO, and Random Forest. The goal was to reduce model complexity and computational overhead without sacrificing predictive accuracy.

Classification Models

The study compared two families of ensemble methods against three baseline classifiers:

  • Baselines: LASSO (linear), k-Nearest Neighbor (distance-based), and SVM (kernel-based).
  • Ensembles: Random Forest (a bagging technique) and XGBoost (a boosting technique).

Dataset Analysis

The study utilized the CSIC2010 v2 dataset, a benchmark for simulating e-commerce web traffic. The original dataset contained 223,585 samples with 18 basic features. Through the authors' refinement process, this was transformed into a more "dense" dataset of 13,569 samples with 78 features, capturing the subtle nuances of how attackers structure malicious payloads compared to normal user behavior.

Key Results and Discussion

The experimental results highlight the superiority of ensemble methods in web traffic classification.

Predictive Performance

Ensemble methods consistently outperformed baseline classifiers by approximately 20% in predictive accuracy. XGBoost achieved the highest performance with a peak AUC of 0.989, followed closely by Random Forest at 0.983. In comparison, the best-performing baseline, SVM, only reached an AUC of 0.866.

Comprehensive performance metrics for various classifiers and feature selection techniques.

Efficiency and Robustness

A critical finding was the "sturdiness" of ensemble models, which showed significantly lower performance variability across different test folds compared to baselines. Furthermore, the study demonstrated a successful efficiency trade-off: using Information Gain for feature selection reduced the variable count from 78 to 29 while maintaining a high AUC of 0.987 with XGBoost. This reduction is vital for real-time systems where processing speed and memory are constrained.

Conclusion

This research demonstrates that combining sophisticated feature engineering with ensemble learning significantly enhances the detection of malicious web traffic. While the aging nature of the CSIC2010 dataset and the prevalence of encrypted HTTPS traffic remain challenges for real-world deployment, the high AUC scores and the effectiveness of dimensionality reduction provide a strong foundation for future security frameworks. Future work will focus on evaluating these models against modern, encrypted traffic datasets and optimizing them for low-latency environments.

Summary updated

BibTeX
@misc{urda2024enhancing,
  author = {Daniel Urda and Branly Martínez and Nuño Basurto and Meelis Kull and Ángel Arroyo and Álvaro Herrero},
  title = {{Enhancing web traffic attacks identification through ensemble methods and feature selection}},
  year = {2024},
  eprint = {2412.16791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.16791},
}
Figure