Abstract
In this paper, a process-aware security task-scheduling algorithm is proposed as IOAware. The algorithm evaluates the performance of hardware in computing nodes and estimates properties of a task while it is being executed. In subsequent assignments, the algorithm assigns tasks based on the performance of the TaskTracker and the task properties. To verify the theoretical feasibility of the proposed IOAware scheduling algorithm, a schedule model is proposed to implement the method. After applying the scheduling module in a Hadoop cluster with multiple experiments, the results show that, compared with the performances of the FIFO, computing capacity-scheduling, and fair-scheduling algorithms, the IOAware scheduling algorithm effectively reduces the execution time of tasks and improves the throughput of the cluster.
The Problem: Heterogeneity and IO Bottlenecks
In large-scale distributed systems like Hadoop, performance often suffers because standard schedulers (like FIFO, Capacity, or Fair) treat computing nodes as relatively uniform. In reality, clusters are often heterogeneous, consisting of hardware with varying CPU speeds and disk performance.
When multiple IO-intensive tasks are assigned to the same node, they compete for disk bandwidth, leaving the CPU idle while waiting for data—a classic IO bottleneck. Conversely, CPU-heavy tasks might leave disk resources underutilized. The IOAware scheduler aims to bridge this gap by understanding the specific resource requirements of a task and matching them to the available hardware.
The IOAware Methodology
The IOAware framework operates in three main phases:
- Hardware Assessment: The algorithm quantitatively estimates the disk performance ($R_{disk}$) of each node by analyzing CPU frequency and execution times of sample tasks.
- Job Classification: Jobs are classified as either CPU-bound or IO-bound. This is determined by calculating the ratio between Map Input Data (MID) and Map Output Data (MOD).
- Dynamic Task Dispatching: The scheduler uses a specific logic flow to assign tasks. If a node is struggling with blocked IO tasks, the scheduler will intentionally assign it a CPU-bound task to maximize resource utilization without further choking the disk.
Experimental Results
The researchers evaluated IOAware using a heterogeneous Hadoop cluster running three standard benchmarks: WordCount (CPU-bound), Grep (CPU-bound), and TeraSort (IO-bound). Through experimentation, the authors identified a threshold value of 0.6 for judging task "obstruction," which provided the optimal balance for minimizing average response time.
When compared against standard scheduling algorithms, IOAware demonstrated superior efficiency. Specifically, it completed the test suite in 1115 seconds, outperforming the FIFO (1123s), Fair (1125s), and Capacity (1135s) schedulers.
Beyond execution time, IOAware showed improved data localization rates and more stable CPU utilization. Visualization of system resources indicated that IOAware produced smoother utilization curves with fewer extreme spikes, suggesting a more predictable and stable cluster environment.
Conclusion
The IOAware algorithm effectively optimizes Hadoop clusters by dynamically matching task resource needs with hardware capabilities. By intelligently co-locating complementary tasks, it reduces disk bottlenecks and increases overall system throughput. Future work may involve extending this process-aware logic to modern architectures like ARM or GPU computing, which are increasingly prevalent in contemporary data centers.
Summary updated