TL;DR
Content Delivery Networks (CDNs) are increasingly used to serve web content, but their actual performance benefits are not well understood. This paper introduces a methodology to measure CDN performance from a client's perspective, finding that while CDNs generally outperform origin servers, the use of short DNS TTLs for load balancing doesn't always translate to better client-perceived response times.
What Problem Does This Paper Address?
The internet has seen a massive growth in web traffic, and a significant portion of this traffic is served by a small number of popular websites. To handle this load and improve user experience, Content Delivery Networks (CDNs) have emerged. CDNs are networks of servers that cache or replicate content from origin servers, aiming to deliver it to users from a server geographically or topologically closer to them. Despite their widespread adoption, there was a lack of published research on how extensively CDNs were being used and how effectively they performed. This paper aims to fill this gap by investigating the usage patterns and performance of CDNs, particularly focusing on how techniques like DNS redirection and URL rewriting impact client-perceived latency.
Methods
The authors developed a client-centric methodology to measure CDN performance. This involved:
- Canonical Page Construction: Creating a standardized web page with a fixed number and size distribution of embedded images, mimicking typical web content. This allowed for consistent comparisons across different CDNs and origin servers.
- Data Collection: Periodically retrieving this canonical page from various CDNs and origin servers using a set of geographically distributed measurement platforms.
- Performance Metrics: Measuring key performance indicators such as DNS lookup time, download time for images, and overall completion time. They tested different HTTP protocols (HTTP/1.0 and HTTP/1.1 with persistent connections and pipelining).
- DNS Load Balancing Analysis: Investigating the effectiveness of CDNs using short DNS Time-To-Live (TTL) values for load balancing. This involved comparing download times when using a newly resolved IP address versus a "fixed" IP address obtained earlier, to assess the benefit of dynamic DNS lookups.
Datasets
- Web Site Lists: The study utilized lists of popular websites, including the "HOTMM127" (Media Metrix top 50 + 100hot.com) and "URL588-MM500" (servers from a previous study + Media Metrix top 500) to identify sites using CDNs.
- CDN Company Data: Information on CDN usage was gathered from a list of 58 sites believed to be served by CDNs other than Akamai (CDN58).
- Proxy Logs: Data from NLANR proxy logs and logs from a large manufacturing company were used to analyze the nature of CDN-served content, particularly focusing on images and streaming media.
- NIMI Measurement Infrastructure: The performance study leveraged approximately two dozen widely deployed measurement "platforms" from the NIMI infrastructure, primarily located at U.S. university and government laboratory sites.
Key Insights and Ideas
- CDN Adoption is Growing Rapidly: The study found a significant increase in CDN usage among popular websites between 1999 and 2000, suggesting CDNs are becoming a mainstream solution for content delivery.
- Images Dominate CDN Content: The vast majority of CDN-served objects are images, though they account for a smaller percentage of the total bytes transferred.
- DNS Redirection's Double-Edged Sword: While DNS redirection is a common CDN technique, the study suggests that frequently forcing DNS lookups with short TTLs doesn't always lead to better client response times. In many cases, using a previously resolved or fixed IP address resulted in comparable or better performance, even without considering the DNS lookup overhead.
- Performance Varies Significantly: The performance of individual CDNs can vary greatly, and a CDN that performs poorly in one test period might excel in another, potentially due to changes in their network infrastructure (e.g., number of servers).
- HTTP/1.1 Offers Improvements: Utilizing HTTP/1.1 with persistent connections and pipelining generally yields better download times compared to parallel HTTP/1.0 requests.
Key Results
- CDN Usage: In December 2000, 31% of sites from one list and 17% from another popular site list used CDNs, a substantial increase from 1-2% in November 1999.
- Content Change: 86-94% of CDN-served objects were seen in previous page retrievals, and less than 1% of the content itself changed, indicating CDNs primarily serve static content.
- Cache Hit Rates: CDN-served images showed 20-30% higher cache hit rates compared to non-CDN-served images.
- Performance Comparison: In general, CDNs provided better download performance than origin servers, especially for larger numbers of images. For instance, in January 2001 tests, CDNs significantly outperformed origin servers.
- DNS TTL Impact: The study found that in many cases, the download time using a newly resolved IP address was not better than using a fixed IP address, and when DNS lookup costs were factored in, the fixed IP address often performed better. This suggests that short DNS TTLs, while intended for load balancing, might not always be beneficial.
- Speedera's Improvement: Speedera showed a dramatic performance improvement between test periods, correlating with a significant increase in its network of servers.
Limitations and Future Work
- Client Site Limitations: The study acknowledges that its client sites were primarily U.S. university and government laboratory sites, representing a narrow client mix. This limits the generalizability of the findings to all user groups.
- Dynamic Nature of CDNs: CDN performance can change rapidly, and the study's findings are snapshots in time. Repeating the study over longer periods is suggested.
- Streaming Media: The paper notes ongoing work to extend the methodology to streaming media content, which, while a smaller fraction of objects, contributes significantly to bytes transferred. This includes investigating protocols like RTSP and encoding formats.
- Broader Conclusions: The authors emphasize that despite the methodology's robustness, drawing broad conclusions about CDN performance is difficult due to the limitations. However, they recommend that content providers use similar studies to evaluate CDN performance for their specific user bases.
Summary updated