A Dataset for Analyzing Streaming Media Performance over HTTP/3 Browsers

Sapna Chaudhary, Mukulika Maity, Sandip Chakraborty, Naval Kumar ShuklaNeurIPS 2023

This research paper, presented at NeurIPS 2023, introduces a significant new resource for the networking and multimedia communities: a large-scale dataset and a specialized collection tool (H3B) designed to dissect how modern browsers handle video streaming over the relatively new HTTP/3 protocol.

TL;DR

The authors introduce H3B, a tool and dataset comprising over 5,400 hours of YouTube streaming data to compare HTTP/3 (QUIC) against HTTP/2 (TCP). Their most striking finding is that the "fallback" mechanism in modern browsers—which switches to older protocols when network conditions are poor—often backfires, resulting in a worse user experience than if the browser had simply stuck with HTTP/3.

What Problem Does This Paper Address?

For decades, the web ran on TCP. HTTP/3 changes the game by using QUIC, a protocol designed to reduce latency and handle network transitions more gracefully. While the industry has rushed to adopt HTTP/3, we lack comprehensive, "in-the-wild" data on how it actually performs in browsers like Chrome and Firefox, especially under the fluctuating network conditions common in the developing world.

Existing datasets are often too small, use simulated video players rather than real production endpoints like YouTube, or fail to capture the crucial "network-level" logs (the actual packets) alongside "application-level" logs (the video quality and stalls). This paper fills that gap, specifically investigating why HTTP/3 doesn't always deliver the performance gains it promises.

Methods

The researchers developed H3B, an emulation-based toolbox that automates YouTube streaming while simultaneously capturing logs from two layers.

Architecture of the H3B tool

As shown in the architecture above, the tool operates across two main layers:

  1. Application Layer: Using browser extensions and YouTube’s "Stats for Nerds," they track bitrates, resolution, and playback buffer health.
  2. Network Layer: Using tcpdump, they capture the raw packet exchange to see exactly which protocol is being used at any given millisecond.

To ensure the results were realistic, they used Mahimahi, a network emulation tool, to replay five different bandwidth patterns, ranging from high-speed connections to "very low" bandwidth and real-world mobility traces collected from cellular users in four different countries. They even went as far as modifying the Chromium source code to create a version of the browser that "forcibly" stays on HTTP/3 to test the impact of protocol switching.

Datasets

The primary contribution is the H3B Dataset:

  • Scale: 5,936 YouTube sessions totaling 5,464 hours of streaming.
  • Diversity: 46 videos across various genres (News, Entertainment, Education).
  • Scope: Data collected across 5 geographical locations (Delhi, Bangalore, New York, Germany, and Singapore).
  • Comparison: The dataset includes "paired" sessions—streaming the same video under the same network conditions once with HTTP/3 enabled and once with it disabled (forcing legacy HTTP/2) for a true head-to-head comparison.

Key Insights and Ideas

  • The "Fallback" Paradox: Browsers are designed to fall back to HTTP/2 over TCP if they detect issues with the QUIC (UDP) path. However, the authors found that browsers often trigger this fallback even when UDP isn't blocked, simply because the application is struggling.
  • HTTP/3 is Not a Guaranteed Winner: In roughly 41% of the cases studied in Chromium, the legacy HTTP/2 protocol actually provided a better Quality of Experience (QoE) than HTTP/3.
  • Implementation Matters More Than Theory: The superiority of QUIC is well-established in theory, but the authors suggest that current browser implementations of "legacy support" and fallback logic might be poorly tuned, leading to unnecessary protocol switching that hurts performance.
  • Geography and Network Type Influence Success: Protocol performance wasn't uniform; for instance, HTTP/3 showed more significant advantages in Bangalore and Germany than in Delhi or Singapore.

Key Results

Performance comparison and TCP fallback behavior
  • QoE Comparison: In 81.5% of sessions, there was a statistically significant difference in performance between the protocols. As seen in the charts above, HTTP/2 frequently outperformed HTTP/3, particularly in Chromium.
  • Protocol Switching Impact: By using their modified Chromium browser (disabling fallback), the researchers found that forcibly staying on HTTP/3 improved QoE in 60% of cases compared to the standard browser behavior.
  • TCP Dominance in Poor Networks: In "Dynamic Very Low" (DVL) bandwidth scenarios, TCP traffic was present 98% of the time during supposed HTTP/3 sessions (illustrated by the CDF plot above), indicating that the browser almost immediately abandoned the newer protocol when things got tough.

Limitations and Future Work

The authors acknowledge a few key areas for growth:

  • Platform Diversity: The current data is limited to desktop browser environments. Extending H3B to mobile apps is a logical next step.
  • Network Variety: While they focused on poor network conditions, adding more high-speed and diverse 5G traces would provide a more complete picture.
  • ML Integration: The authors suggest this dataset is prime territory for training machine learning models to perform "intelligent protocol selection"—essentially teaching the browser when to stick with HTTP/3 and when to switch, rather than relying on the current, somewhat flawed, heuristics.

Check out the project: The tool and dataset are available on GitHub.

Summary updated

BibTeX
@inproceedings{chaudhary2023dataset,
  author = {Sapna Chaudhary and Mukulika Maity and Sandip Chakraborty and Naval Kumar Shukla},
  title = {{A Dataset for Analyzing Streaming Media Performance over HTTP/3 Browsers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure