The Automatic Statistician

Christian Steinruecken, Emma Smith, David Janz, James Robert Lloyd, Zoubin GhahramaniAutomated Machine Learning 2019 · OpenReview

Abstract

The Automatic Statistician project aims to automate data science, producing predictions and human-readable reports from raw datasets with minimal human intervention. Alongside basic graphs and statistics, the generated reports contain a curation of high-level insights about the dataset that are obtained from (1) an automated construction of models for the dataset, (2) a comparison of these models, and (3) a software component that turns these results into natural language descriptions. This chapter describes the common architecture of such Automatic Statistician systems, and discusses some of the design decisions and technical challenges.

Introduction

Despite the rapid advancement of Artificial Intelligence, machine learning remains a "cottage industry" heavily dependent on human experts. These experts must manually clean data, select features, and tune models—a process that is slow, expensive, and prone to bias. Furthermore, many state-of-the-art models are "black boxes." While accurate, they are difficult to interpret, debug, or trust in high-stakes environments like medicine or autonomous driving. The Automatic Statistician project addresses these challenges by creating a system that can reason about data patterns and explain them to non-experts in plain English.

System Architecture

The core of the system is built on model-based machine learning using Gaussian Processes (GPs). The architecture follows a pipeline that transforms raw data into a comprehensive, interpretable report.

System Architecture Overview

The process begins with a "Language of Models" that defines the search space. A search procedure explores this space, evaluated by a checking component, ultimately producing a model that generates predictions and natural language explanations.

The Language of Models: Compositional Grammar

Instead of using a single fixed model, the system employs a compositional grammar over kernels. It starts with base kernels representing simple patterns:

  • Constant (C): Captures the mean level.
  • Linear (LIN): Captures trends.
  • Squared Exponential (SE): Captures smooth variations.
  • Periodic (PER): Captures repeating patterns.
  • White Noise (WN): Captures random fluctuations.

These base kernels are combined using operators like addition (+), multiplication (x), and change-points (CP) to create complex, expressive models.

The Automatic Statistician Pipeline

Automated Search and Evaluation

The system explores the infinite space of possible models using a greedy search strategy. It starts with simple kernels and iteratively adds or mutates them to find the most promising structure. To avoid overfitting, the system employs the Bayesian Information Criterion (BIC). This acts as a "Bayesian Occam’s Razor," penalizing overly complex models unless they provide a significantly better fit to the data.

Natural Language Reporting

Once the best-fitting model is identified, the system translates the mathematical kernel expression into a canonical form and then into human-readable English. For example, a complex kernel might be described as "a smooth function modulated by a periodic function with linearly varying amplitude." This ensures that the discovered patterns are accessible to non-experts.

Empirical Findings

The system's capabilities have been demonstrated primarily on time-series data:

  • Sunspot Activity: The system automatically identified the ~11-year solar cycle and historical anomalies like the Maunder Minimum (a period of low sunspot activity in the 17th century) purely from raw data.
  • Human-Like Intuition: In extrapolation studies, the system's predictions were compared against human reasoning. Participants often preferred the outputs of the Automatic Statistician over standard, non-compositional machine learning models, suggesting the AI's "intuition" aligns well with human perception of patterns.

Conclusion and Future Work

The Automatic Statistician represents a significant step toward interpretable, automated AI. By restricting the model space to a grammar of kernels with human-language equivalents, the system ensures transparency by construction.

However, challenges remain. The current greedy search may miss global optima, and the O(N^3) computational complexity of Gaussian Processes limits scalability to massive datasets. Future work aims to incorporate approximation techniques for "Big Data" and develop interactive dialogue systems that allow users to guide the model discovery process.

Summary updated

Abstract

The Automatic Statistician project aims to automate data science, producing predictions and human-readable reports from raw datasets with minimal human intervention. Alongside basic graphs and statistics, the generated reports contain a curation of high-level insights about the dataset that are obtained from (1) an automated construction of models for the dataset, (2) a comparison of these models, and (3) a software component that turns these results into natural language descriptions. This chapter describes the common architecture of such Automatic Statistician systems, and discusses some of the design decisions and technical challenges.
BibTeX
@inproceedings{steinruecken2019automatic,
  author = {Christian Steinruecken and Emma Smith and David Janz and James Robert Lloyd and Zoubin Ghahramani},
  title = {{The Automatic Statistician}},
  booktitle = {Automated Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=ieMyyIPFrY},
}
Figure