Multiagent Router Throttling: Decentralized Coordinated Response Against DDoS Attacks

Kleanthis Malialis, Daniel Kudenko

TL;DR

This paper introduces Multiagent Router Throttling (MRT), a novel decentralized system where upstream routers learn to collaboratively throttle traffic to defend against DDoS attacks. MRT offers a more secure, reliable, and cost-effective solution compared to existing server-initiated methods, demonstrating superior or comparable performance across various attack scenarios.

What Problem Does This Paper Address?

Distributed Denial of Service (DDoS) attacks are a significant threat to internet availability, overwhelming servers and network infrastructure with massive traffic floods. Existing defense mechanisms, like Router Throttling, often rely on a server initiating commands to upstream routers. This creates a single point of failure, as the server itself can be targeted, or a helper machine might be needed, which can also be attacked. Furthermore, these methods can lead to collateral damage by indiscriminately throttling legitimate traffic. The paper aims to address these limitations by proposing a decentralized, multiagent approach where routers learn to adaptively throttle traffic without direct server command, enhancing security and efficiency.

Methods

The core of the proposed solution is Multiagent Router Throttling (MRT). This system utilizes multiple independent reinforcement learning (RL) agents, each installed on an upstream router. These agents learn to probabilistically drop incoming traffic destined for a victim server.

Here's a breakdown of the key methodological aspects:

  • Decentralized Architecture: Unlike server-initiated methods, each router agent makes its own decisions based on its local observations and learned policy. This eliminates the single point of failure associated with a centralized controller.
  • Reinforcement Learning: The agents are trained using a Markov Decision Process (MDP) framework. Each agent's state is its average load over a monitoring window. The action space involves choosing a drop probability (e.g., 0.4 means dropping 40% of traffic).
  • Reward Function: The system has two primary goals encoded in the reward function:
  • Keep the server operational: A penalty (-1) is given if the server load exceeds an upper boundary ($U_s$).
  • Maximize legitimate traffic: A reward ($L \in [0, 1]$) is given based on the proportion of legitimate traffic that reaches the server during a time step.
  • Learning Algorithm: The paper employs the SARSA algorithm, a common temporal-difference learning method. For exploration, a linearly decreasing ε-greedy strategy is used.
  • Agent Selection: Agents are installed on routers that are within a certain hop distance ($k$) from the server, or directly attached to hosts. This ensures that the selected routers are positioned to intercept significant portions of the traffic.
  • Simulation Environment: The ns-2 network simulator is used for training and evaluation.

Datasets

The paper does not use pre-existing datasets in the traditional sense. Instead, it relies on:

  • Simulated Network Environments: The authors create synthetic network topologies and traffic patterns within the ns-2 simulator for both training and evaluation.
  • Training Scenarios: During training, the system is exposed to a dynamic environment where network instances are regenerated in each episode. This involves randomly selecting legitimate users, attackers, and their sending rates according to specified probabilities.
  • Evaluation Scenarios: For evaluation, the system is tested against six distinct attack dynamics: constant rate, increasing rate, and three types of pulse attacks (long, medium, short), as well as a group attack. These scenarios are designed to be more challenging and realistic, including using legitimate packet interarrival times that follow a Pareto distribution, which differs from the training setup.

Key Insights and Ideas

  • Decentralization is Key for Resilience: The paper strongly advocates for decentralized defense mechanisms. By distributing the decision-making to individual routers, the system becomes inherently more robust against attacks that target central points.
  • RL Agents Adapt to Dynamic Threats: The multiagent RL approach allows routers to learn and adapt their throttling strategies in response to evolving and sophisticated DDoS attack patterns. This is crucial because DDoS tactics are constantly changing.
  • Balancing Server Availability and Legitimate Traffic: The carefully designed reward function is critical. It forces the agents to not only protect the server but also to minimize the impact on legitimate users, a common challenge in DDoS mitigation.
  • "Meek" Attackers Pose a Unique Challenge: The paper acknowledges that differentiating between legitimate and attack traffic becomes difficult when attackers use rates similar to legitimate users. This highlights a limitation and suggests future research directions.
  • Training on Diverse Scenarios is Crucial: The authors emphasize that training the system across a wide range of network instances and attack variations is essential for developing a robust and generalizable policy.

Key Results

The evaluation results demonstrate the effectiveness of the Multiagent Router Throttling (MRT) approach:

  • Superior Performance: In all tested scenarios, MRT outperforms the Baseline Router Throttling approach (statistically significant differences).
  • Competitive with AIMD: MRT shows comparable performance to the AIMD Router Throttling approach in scenarios involving constant rate, increasing rate, and long pulse attacks.
  • Outperforms AIMD in Complex Scenarios: MRT outperforms the AIMD approach in scenarios involving medium pulse, short pulse, and group attacks.
  • Notable Failure of AIMD: In the case of short pulse attacks (where the pulse period matches the monitoring window), the AIMD approach performs worse than even the Baseline.
  • Security, Reliability, and Cost-Effectiveness: The paper claims MRT is more secure (no single point of failure), reliable (no need for complex communication protocols), and cost-effective (no authentication, priority transmission, or retransmission needed) compared to existing methods.

Limitations and Future Work

  • "Meek" Attackers: The current approach struggles to differentiate between legitimate and attack traffic when attackers use rates similar to legitimate users. This limitation suggests the need to enrich the agent's state space with more statistical features beyond local router load.
  • Scalability: The paper acknowledges scalability as an important challenge and plans to investigate how MRT can be extended to incorporate a larger number of learning agents.
  • Network Model Accuracy: The training phase relies on distinguishing between legitimate and attack traffic, which is difficult in real-world scenarios. The authors suggest exploring online learning without a model to address this.
  • Exploration Strategy: While the current approach uses a decreasing ε-greedy strategy, further investigation into more advanced exploration techniques could be beneficial.

Summary updated

Abstract

Distributed denial of service (DDoS) attacks constitute a rapidly evolving threat in the current Internet. In this paper we introduce Multiagent Router Throttling, a decentralized DDoS response mechanism in which a set of upstream routers independently learn to throttle traffic towards a victim server. We compare our approach against a baseline and a popular throttling technique from the literature, and we show that our proposed approach is more secure, reliable and cost-effective. Furthermore, our approach outperforms the baseline technique and either outperforms or has the same performance as the popular one.
BibTeX
@inproceedings{malialis2013multiagent,
  author = {Kleanthis Malialis and Daniel Kudenko},
  title = {{Multiagent Router Throttling: Decentralized Coordinated Response Against DDoS Attacks}},
  booktitle = {Iaai},
  year = {2013},
  doi = {10.1609/aaai.v27i2.19000},
}
Figure