HAPI Explorer: Comprehension, Discovery, and Explanation on History of ML APIs

Lingjiao Chen, Zhihua Jin, Sabri Eyuboglu, Huamin Qu, Christopher Ré, Matei Zaharia, James Zou

Machine learning APIs from providers like Google and Amazon evolve constantly, but tracking their performance shifts over time is notoriously difficult. HAPI Explorer is an interactive, web-based visualization tool that allows users to analyze the historical performance, consistency, and biases of commercial ML APIs. By offering global, subpopulation, and instance-level views, it helps developers make informed decisions about which APIs to trust and how to adapt their machine learning pipelines.

The Challenge of Evolving APIs

Developers increasingly rely on Machine Learning as a Service (MLaaS) platforms for complex tasks like visual object detection and natural language comprehension. However, these APIs are not static; providers continuously update their models behind the scenes. This creates a "black box" evolution where an API's overall accuracy, prediction consistency, and fairness might shift over time without warning. Systematically studying these changes is difficult because researchers and engineers lack both the longitudinal data and the user-friendly exploration tools needed to track API behavior over months or years. HAPI Explorer was built to bridge this gap.

The HAPI Explorer Framework

HAPI Explorer provides a programming-free, interactive visual interface to track API dynamics. The system is structured around a "zoom in/zoom out" workflow, offering three levels of analysis:

The three-level workflow of HAPI Explorer: (a) Global Analysis, (b) User-defined Regime Exploration, and (c) Data Instance Search and Visualization.

1. Global Performance Analysis

This level visualizes high-level metrics across different datasets and APIs over time. It tracks standard accuracy, prediction consistency (how often an API gives the same prediction year-over-year), confidence shifts, and group disparity (fairness). It uses line charts for performance and triangle plots to map consistency rates between different years.

2. User-Defined Local Regime Exploration

Users can filter and query specific subpopulations using parallel sets—a visualization paradigm that turns a dataset into a multi-dimensional flow. This allows users to see exactly how prediction confidence scores for a specific subset of data shift from one year to the next, revealing hidden biases or sudden drops in consistency.

Developers can drill down to a single data point (e.g., a specific image). Users can search by ground truth labels and see exactly how different APIs scored that specific instance across different years.

Instance-level visualization showing how different models predicted a specific 'aeroplane' image over time, including confidence scores and predicted labels.

The HAPI Dataset

The system is built on top of the HAPI (History of API) dataset, a longitudinal database introduced in 2022. It contains over 1 million instances of commercial API applications collected over three years, covering major tech companies across a range of tasks. HAPI Explorer also allows users to upload and analyze their own custom collections of API instances.

Key Insights and Results

  • APIs are dynamic, not static: Treating a commercial ML API as a fixed mathematical function is a risky engineering practice. Their performance and confidence levels shift over time, which can silently break downstream applications that rely on strict confidence thresholds.
  • Granularity matters: Global accuracy metrics can hide critical flaws. By allowing users to "zoom in" from global trends to specific subpopulations, developers can uncover hidden biases that only affect certain types of data.
  • Visualizing confidence flows: Using parallel sets to visualize how an API's confidence on the exact same data changes year-over-year provides an intuitive way to spot model drift or underlying architecture changes.

As a system demonstration, HAPI Explorer successfully processes and visualizes over 1 million API interactions, providing real-time, interactive filtering for accuracy, consistency, and fairness. The interface allows users to easily compare multiple APIs side-by-side to determine which provider offers the most stable and accurate service for their specific data niche.

Future Directions

The release of HAPI Explorer aims to stimulate further research into the dynamics of ML APIs. By making historical data accessible, the authors hope to help developers build more robust, adaptive ML pipelines that can intelligently handle silent API updates.

Summary updated

Abstract

Machine learning prediction APIs offered by Google, Microsoft, Amazon, and many other providers have been continuously adopted in a plethora of applications, such as visual object detection, natural language comprehension, and speech recognition. Despite the importance of a systematic study and comparison of different APIs over time, this topic is currently under-explored because of the lack of data and user-friendly exploration tools. To address this issue, we present HAPI Explorer (History of API Explorer), an interactive system that offers easy access to millions of instances of commercial API applications collected in three years, prioritize attention on user-defined instance regimes, and explain interesting patterns across different APIs, subpopulations, and time periods via visual and natural languages. HAPI Explorer can facilitate further comprehension and exploitation of ML prediction APIs.
BibTeX
@inproceedings{chen2023hapi,
  author = {Lingjiao Chen and Zhihua Jin and Sabri Eyuboglu and Huamin Qu and Christopher Ré and Matei Zaharia and James Zou},
  title = {{HAPI Explorer: Comprehension, Discovery, and Explanation on History of ML APIs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27064},
}
Figure