Secure Communication Channel Establishment: TLS 1.3 (over TCP Fast Open) versus QUIC

Shan Chen, Samuel Jero, Matthew Jagielski, Alexandra Boldyreva, Cristina Nita-RotaruJ. Cryptol 2021 · DOI · OpenReview

Abstract

This paper, published in the Journal of Cryptology, provides the first comprehensive "layered" security comparison between TLS 1.3 (running over TCP Fast Open) and QUIC. While both protocols aim to minimize latency through "0-RTT" (zero round-trip time) handshakes, their security profiles differ significantly at the transport-layer. Using a new formal framework, msACCE, the authors demonstrate that TLS 1.3 is more vulnerable to packet-level availability attacks due to unauthenticated transport headers. In contrast, the integrated design of QUIC[TLS] emerges as the most robust overall architecture.

Introduction

For decades, internet communication relied on a sequential handshake: TCP established a connection, followed by TLS establishing security. To minimize latency, modern protocols like TLS 1.3 and QUIC have moved toward 0-RTT handshakes, where data is sent in the initial packet. However, traditional security analysis often ignores the interaction between cryptographic layers and transport layers, creating a blind spot regarding transport-level vulnerabilities. This research addresses this gap by providing a formal "layered" security comparison between TLS 1.3 (over TCP Fast Open) and QUIC.

Methods and the msACCE Framework

The authors introduce a new formal protocol syntax called msACCE (multi-stage Authenticated and Confidential Channel Establishment). This model handles protocols that establish multiple keys at different stages, such as the transition from a 0-RTT key to a more secure 1-RTT key.

Extracted figure from page 8 of Secure Communication Channel Establishment: TLS 1.3 (over TCP Fast Open) versus QUIC

The study evaluates three specific protocol stacks: TFO + TLS 1.3, Google QUIC, and the IETF standardized QUIC[TLS]. Protocols are evaluated using two models: msACCE-std, focusing on standard goals like server authentication and confidentiality, and msACCE-pauth, a novel model for packet-authentication that assesses availability goals, such as resistance to header manipulation or forged reset packets.

Key Insights: The Layering Penalty

A primary finding is the "layering penalty" inherent in the TLS 1.3 over TCP stack. Because TLS treats TCP as a "black box," it does not authenticate the TCP headers. This lack of integration allows an attacker to manipulate unencrypted fields to disrupt service even if they cannot break the encryption itself.

The standard TCP header. Fields like Window and Sequence Number are not authenticated by TLS, leading to potential availability attacks.

In contrast, QUIC[TLS] integrates transport and cryptography into a single layer, allowing it to authenticate its own headers. This architectural difference makes QUIC significantly more robust against packet-level availability attacks.

Results and Security Comparison

The analysis yields several critical results regarding protocol robustness:

  • Header Integrity: TFO+TLS 1.3 fails to provide header integrity. Attackers can modify TCP headers, such as the window field, to stall connections. QUIC and QUIC[TLS] pass this test because they authenticate their headers.
  • Reset Authentication: Only QUIC[TLS] provides a secure mechanism for stateless resets. In the TFO+TLS 1.3 stack, an attacker can easily forge a TCP RST packet to terminate a healthy session.
  • IP Spoofing: All three protocols are secure against IP spoofing, though they use different mechanisms (TFO cookies vs. QUIC source-address tokens). The paper provides the first formal proof that TFO’s cookie mechanism is effective if the generation function is a cryptographically strong pseudorandom function (PRF).
  • 0-RTT Vulnerabilities: All protocols face challenges with "KE Payload Integrity." Attackers can often manipulate early 0-RTT packets to trigger a "ServerReject," forcing a downgrade to a slower 1-RTT mode—a subtle but effective denial-of-service attack.

Conclusion

The research concludes that while TLS 1.3 and QUIC are both cryptographically sound, their transport-layer implementations lead to divergent security profiles. QUIC[TLS] emerges as the most secure overall protocol due to its ability to authenticate transport headers and provide secure stateless resets. Future work may extend these models to cover mutual authentication and the complex multiplexed data streams characteristic of QUIC.

Summary updated

Abstract

Secure channel establishment protocols such as Transport Layer Security (TLS) are some of the most important cryptographic protocols, enabling the encryption of Internet traffic. Reducing latency (the number of interactions between parties before encrypted data can be transmitted) in such protocols has become an important design goal to improve user experience. The most important protocols addressing this goal are TLS 1.3, the latest TLS version standardized in 2018 to replace the widely deployed TLS 1.2, and Quick UDP Internet Connections (QUIC), a secure transport protocol from Google that is implemented in the Chrome browser. There have been a number of formal security analyses for TLS 1.3 and QUIC, but their security, when layered with their underlying transport protocols, cannot be easily compared. Our work is the first to thoroughly compare the security and availability properties of these protocols. Toward this goal, we develop novel security models that permit “layered” security analysis. In addition to the standard goals of server authentication and data confidentiality and integrity, we consider the goals of IP spoofing prevention, key exchange packet integrity, secure channel header integrity, and reset authentication, which capture a range of practical threats not usually taken into account by existing security models that focus mainly on the cryptographic cores of the protocols. Equipped with our new models we provide a detailed comparison of three low-latency layered protocols: TLS 1.3 over TCP Fast Open (TFO), QUIC over UDP, and QUIC[TLS] (a new design for QUIC that uses TLS 1.3 key exchange) over UDP. In particular, we show that TFO’s cookie mechanism does provably achieve the security goal of IP spoofing prevention. Additionally, we find several new availability attacks that manipulate the early key exchange packets without being detected by the communicating parties. By including packet-level attacks in our analysis, our results shed light on how the reliability, flow control, and congestion control of the above layered protocols compare, in adversarial settings. We hope that our models will help protocol designers in their future protocol analyses and that our results will help practitioners better understand the advantages and limitations of secure channel establishment protocols.
BibTeX
@article{chen2021secure,
  author = {Shan Chen and Samuel Jero and Matthew Jagielski and Alexandra Boldyreva and Cristina Nita-Rotaru},
  title = {{Secure Communication Channel Establishment: TLS 1.3 (over TCP Fast Open) versus QUIC}},
  journal = {J. Cryptol.},
  year = {2021},
  doi = {10.1007/s00145-021-09389-w},
}

Loading related content...

Figure