Mixed Messages: Measuring Conformance and Non-Interference in TypeScript (Artifact)

Jack Williams, J. Garrett Morris, Philip Wadler, Jakub ZalewskiDagstuhl Artifacts Ser 2017 · OpenReview

TL;DR

This paper introduces an artifact for reproducing experiments on gradual typing in TypeScript. It allows researchers to re-evaluate the effectiveness of TypeScript's type definitions and the underlying JavaScript proxy mechanism for enforcing type contracts, highlighting significant issues with both.

What Problem Does This Paper Address?

As JavaScript continues to dominate web development, the need for robust type safety has become paramount. Gradual typing, which allows for a mix of statically typed and dynamically typed code, offers a potential solution. TypeScript is a popular language that implements gradual typing by allowing developers to write type annotations for JavaScript code. However, the effectiveness of these type definitions and the mechanisms used to enforce them in a gradual typing setting are not fully understood. This paper addresses the question of whether TypeScript's type definitions are accurate and whether the JavaScript proxy mechanism, often used for runtime contract enforcement, is a reliable tool for gradual typing.

Methods

The core of this work is the TypeScript TPD tool, which is based on the polymorphic blame calculus. This tool is designed to monitor JavaScript libraries and their corresponding TypeScript definition files. It essentially acts as a runtime checker, comparing the actual behavior of JavaScript code against the expected behavior defined by TypeScript. The artifact provides the source code for this tool, allowing others to compile and execute it.

Datasets

The artifact includes 122 JavaScript libraries that were tested. For each library, the corresponding TypeScript definition files are also provided. These libraries and their definitions, along with the tool itself, form the basis for reproducing the experiments and the log data presented in the original paper. The artifact also contains pre-recorded examples of type errors and interference that were discovered during the original research.

Key Insights and Ideas

  • TypeScript Definitions are Error-Prone: A significant finding is that the type definitions provided for JavaScript libraries are not always accurate. This means that even when code appears to be correctly typed according to TypeScript, it can still lead to runtime errors.
  • JavaScript Proxies Have Technical Concerns: The paper identifies serious technical issues with using JavaScript's built-in proxy mechanism for enforcing gradual typing contracts. This suggests that this common approach may not be as robust as initially assumed.
  • Reproducibility is Key: The primary contribution of this artifact is to enable the reproduction of the original findings. By providing the tool, libraries, and data, the authors allow the research community to independently verify their conclusions.

Key Results

The artifact allows for the reproduction of conformance errors and examples of interference. While specific quantitative results like accuracy percentages are not detailed in this artifact description, the core outcome is the ability to generate logs that match the tables presented in the original paper, thereby validating the two key insights mentioned above.

Limitations and Future Work

The artifact is provided as a VirtualBox image running Ubuntu 16.04, which might be a limitation for users not familiar with or able to run VirtualBox. The authors acknowledge that the artifact is intended to reproduce their findings, and further work could explore more extensive testing or alternative enforcement mechanisms. The paper itself suggests that the technical concerns with JavaScript proxies warrant further investigation into more reliable contract enforcement strategies for gradual typing.

Summary updated

Abstract

In the paper Mixed Messages: Measuring Conformance and Non-Interference in TypeScript we present our experiences of evaluating gradual typing using our tool TypeScript TPD. The tool, based on the polymorphic blame calculus, monitors JavaScript libraries and TypeScript clients against the corresponding TypeScript definition. Our experiments yield two conclusions. First, TypeScript definitions are prone to error. Second, there are serious technical concerns with the use of the JavaScript proxy mechanism for enforcing contracts. This artifact includes all the libraries we tested, their definition files, and the source code of our tool. From this, all libraries can be wrapped and tested to reproduce the log data that formed our conclusion. All conformance errors and examples of interference are documented, and can be verified against the generated logs.
BibTeX
@inproceedings{williams2017mixed,
  author = {Jack Williams and J. Garrett Morris and Philip Wadler and Jakub Zalewski},
  title = {{Mixed Messages: Measuring Conformance and Non-Interference in TypeScript (Artifact)}},
  booktitle = {Dagstuhl Artifacts Ser},
  year = {2017},
  url = {https://openreview.net/forum?id=qHoK1R1cDD},
}
Figure