Machine learning APIs from providers like Google and Amazon evolve constantly, but tracking their performance shifts over time is notoriously difficult. HAPI Explorer is an interactive, web-based visualization tool that allows users to analyze the historical performance, consistency, and biases of commercial ML APIs. By offering global, subpopulation, and instance-level views, it helps developers make informed decisions about which APIs to trust and how to adapt their machine learning pipelines.
The Challenge of Evolving APIs
Developers increasingly rely on Machine Learning as a Service (MLaaS) platforms for complex tasks like visual object detection and natural language comprehension. However, these APIs are not static; providers continuously update their models behind the scenes. This creates a "black box" evolution where an API's overall accuracy, prediction consistency, and fairness might shift over time without warning. Systematically studying these changes is difficult because researchers and engineers lack both the longitudinal data and the user-friendly exploration tools needed to track API behavior over months or years. HAPI Explorer was built to bridge this gap.
The HAPI Explorer Framework
HAPI Explorer provides a programming-free, interactive visual interface to track API dynamics. The system is structured around a "zoom in/zoom out" workflow, offering three levels of analysis:
1. Global Performance Analysis
This level visualizes high-level metrics across different datasets and APIs over time. It tracks standard accuracy, prediction consistency (how often an API gives the same prediction year-over-year), confidence shifts, and group disparity (fairness). It uses line charts for performance and triangle plots to map consistency rates between different years.
2. User-Defined Local Regime Exploration
Users can filter and query specific subpopulations using parallel sets—a visualization paradigm that turns a dataset into a multi-dimensional flow. This allows users to see exactly how prediction confidence scores for a specific subset of data shift from one year to the next, revealing hidden biases or sudden drops in consistency.
3. Efficient Individual Instance Search
Developers can drill down to a single data point (e.g., a specific image). Users can search by ground truth labels and see exactly how different APIs scored that specific instance across different years.
The HAPI Dataset
The system is built on top of the HAPI (History of API) dataset, a longitudinal database introduced in 2022. It contains over 1 million instances of commercial API applications collected over three years, covering major tech companies across a range of tasks. HAPI Explorer also allows users to upload and analyze their own custom collections of API instances.
Key Insights and Results
- APIs are dynamic, not static: Treating a commercial ML API as a fixed mathematical function is a risky engineering practice. Their performance and confidence levels shift over time, which can silently break downstream applications that rely on strict confidence thresholds.
- Granularity matters: Global accuracy metrics can hide critical flaws. By allowing users to "zoom in" from global trends to specific subpopulations, developers can uncover hidden biases that only affect certain types of data.
- Visualizing confidence flows: Using parallel sets to visualize how an API's confidence on the exact same data changes year-over-year provides an intuitive way to spot model drift or underlying architecture changes.
As a system demonstration, HAPI Explorer successfully processes and visualizes over 1 million API interactions, providing real-time, interactive filtering for accuracy, consistency, and fairness. The interface allows users to easily compare multiple APIs side-by-side to determine which provider offers the most stable and accurate service for their specific data niche.
Future Directions
The release of HAPI Explorer aims to stimulate further research into the dynamics of ML APIs. By making historical data accessible, the authors hope to help developers build more robust, adaptive ML pipelines that can intelligently handle silent API updates.
Summary updated