TL;DR
This paper introduces new middleware for data-intensive applications on high-speed wide-area networks. They demonstrate a composable UDP-based transport protocol (Composible-UDT) and a streaming data mining algorithm (BESH) that together enable efficient analysis of massive datasets across geographically distributed locations. This work is crucial for unlocking the potential of modern high-performance networks for scientific discovery and data analysis.
What Problem Does This Paper Address?
The advent of high-speed (1Gb/s and 10Gb/s) wide-area networks presents a tremendous opportunity for data-intensive applications. However, existing infrastructure and software are not well-equipped to leverage these capabilities. Traditional approaches often involve minimizing data communication, resorting to physically shipping data, or suffer from significant delays when analyzing distributed datasets. Furthermore, the standard Internet transport protocol, TCP, is known to underutilize the bandwidth of high-speed, long-distance networks. Data mining software also typically isn't designed to handle the sheer volume and speed of data flowing over these networks. This paper addresses the challenge of building and deploying data-intensive applications that can effectively utilize these high-performance networks for distributed data analysis.
Methods
The authors developed two key pieces of middleware:
- Composible-UDT: This is an enhanced version of the UDT (UDP-based Data Transfer) protocol. UDT itself uses UDP for data transfer but implements its own reliability and congestion control mechanisms, making it suitable for high-speed networks. Composible-UDT adds a framework that allows for the easy implementation and switching of different congestion control algorithms, making it more adaptable.
- UDT-Gateway: To bridge the gap between existing TCP-based applications and the new UDT protocol, UDT-Gateway was created. It acts as an HTTP file server, allowing users to access data via standard web protocols. However, behind the scenes, it streams data from a remote server using UDT over the high-performance network, only using TCP for the final "last mile" connection to the end-user. This makes UDT-based data services accessible to a wider range of applications and users.
- BESH (Best Effort Streaming Histogram): This is a novel data mining middleware designed for streaming data. It processes data in a single pass, requiring only a fixed amount of storage regardless of the stream's size. BESH dynamically splits and merges histogram buckets to adapt to evolving data distributions. It's designed to handle high-volume data flows, a critical requirement for modern network applications.
Datasets
The paper describes experiments using two primary datasets:
- Sloan Digital Sky Survey (SDSS) Data: This is a massive astronomical dataset, with the BESTDR3 release comprising 797 GB of compressed data (expanding to 1.5 TB when uncompressed). This dataset was used to demonstrate the transfer of large scientific datasets and for analyzing star brightness distributions.
- World Cup 98 Web Log Data: This dataset was used to demonstrate the streaming histogram algorithm. It consisted of web log data from four web servers, with each record containing attributes like the number of bytes transferred.
Key Insights and Ideas
- UDT's Composability is Key: The ability to easily swap congestion control algorithms in Composible-UDT is highlighted as a significant advantage for adapting to diverse network conditions.
- Bridging the Gap with UDT-Gateway: The UDT-Gateway provides a practical solution for integrating high-performance UDT transfers into existing workflows that rely on standard protocols like HTTP/TCP. This significantly broadens the applicability of UDT.
- Streaming Histograms for Big Data: BESH demonstrates that it's possible to perform meaningful data mining (histogram computation) on massive, high-speed data streams in near real-time, even with limited memory.
- Network Performance is Achievable: The demonstrations show that the developed middleware can achieve throughputs close to the hardware limits (e.g., >7 Gb/s for memory-to-memory transfers, and aggregate streaming data mining throughputs reaching 8 Gb/s with peaks of 14 Gb/s).
- Disk I/O as a Bottleneck: In some large-scale transfers, the speed of disk input/output became a significant limiting factor, underscoring the need for balanced system performance.
Key Results
- SDSS Data Transfer: The UDT-Gateway was used to transfer 797 GB of SDSS data from San Diego to Daejeon (Korea) in approximately 2.5 hours, achieving an average transfer speed of 1027 Mb/s. This was noted as a significant achievement for disk-to-disk transfers across the Pacific.
- Streaming Histogram Performance (US109): The BESH algorithm processed four high-volume data streams (web logs) with an average aggregate throughput of around 8 Gb/s, peaking at 14 Gb/s. Per-flow throughputs averaged 2 Gb/s.
- SDSS Brightness Analysis (Post-iGrid): After iGrid, the BESH algorithm was applied to SDSS data transferred from Chicago to Tokyo over a 10Gb/s link. This resulted in an average throughput of 3 Gb/s for the data transfer and processing.
- BESH Accuracy: The paper shows that the BESH algorithm, when applied to web traffic data, produced histograms that were nearly identical to those generated by a standard, non-streaming algorithm, indicating high accuracy.
Limitations and Future Work
- Disk I/O Bottleneck: The paper acknowledges that disk I/O speed was a major bottleneck in some of the large-scale data transfers, suggesting that optimizing storage performance is crucial for fully realizing the potential of high-speed networks.
- Composible-UDT as an Ongoing Project: The authors mention that Composible-UDT is still under development, with future releases planned to include more configuration abilities, such as limited data reliability.
- Generalizability of BESH: While BESH performed well on the tested datasets, further research could explore its performance and applicability to a wider range of data types and distributions. The paper also notes that more detailed analysis of histograms and streaming data processing is beyond its scope.
Summary updated