Automatic Discovery of the Statistical Types of Variables in a Dataset

Isabel Valera, Zoubin GhahramaniICML 2017 · OpenReview

Most machine learning pipelines assume we already know if a variable is categorical, ordinal, or real-valued. However, real-world data is often undocumented and messy. This paper introduces a Bayesian method that automatically discovers the most likely statistical type for every variable in a dataset by modeling them as a mixture of different likelihood functions tied together by a latent low-rank structure.

The Problem with "Messy" Data

In the clean world of textbook statistics, data types are unambiguous: age is a count, salary is a positive real, and eye color is categorical. In the real world of data science, we often inherit massive, poorly documented tables where these distinctions are blurred.

Choosing the wrong data type—like treating an ordinal variable (e.g., "low," "medium," "high") as purely categorical—discards valuable structural information. Conversely, assuming a variable is Gaussian when it is actually bounded (interval data) can lead to poor model fit. While simple heuristics like counting unique values can help, they fail to distinguish between subtle cases, such as whether a continuous variable has an infinite tail or is restricted to a specific interval. The authors aim to automate this "data wrangling" step using a principled probabilistic framework.

Methods: A Generative Model for Data Types

The authors propose a generative model where the observed data is explained by a latent low-rank representation. The core innovation lies in how they handle the likelihood of each attribute.

Graphical model illustrating the pseudo-observation approach and latent structure.
  1. Mixture of Likelihoods: Instead of picking one strict type, they model each attribute as a mixture of several potential likelihood models (e.g., Gaussian for real-valued, Probit for categorical, etc.). The goal of the inference is to find the "weights" for these mixtures, which effectively tells you the probability of an attribute belonging to a specific type.
  2. The "Pseudo-Observation" Trick: To handle heterogeneous data types (continuous and discrete) in a unified way, they use an auxiliary variable approach. They assume there is a hidden Gaussian "pseudo-observation" that is mapped to the actual observed value via a transformation function. For continuous data, this function might be an identity or log-function; for discrete data, it acts as a thresholder mapping regions of the real line to categories or counts.
  3. Inference: They derive an efficient Markov Chain Monte Carlo (MCMC) algorithm to jointly infer the latent features, the weighting vectors, and the likelihood weights.

Experimental Evaluation

The authors evaluated their method on both controlled synthetic scenarios and real-world datasets from the UCI Machine Learning Repository (including Abalone, Adult, Chess, Dermatology, German Credit, Student Performance, and Wine). These datasets were chosen because they contain a complex mix of discrete, continuous, and binary attributes.

Key Insights and Ideas

  • Latent Structure as a Bridge: By assuming a low-rank structure, the model uses the correlations between variables to help identify the type of an individual variable. If a variable behaves like other known ordinal variables in the latent space, the model is more likely to label it as ordinal.
  • The Finite Data Ambiguity: One major insight from the synthetic tests is the realization that with finite data, it is mathematically difficult to distinguish between certain types.
Inferred weights for synthetic Gamma-distributed data.

For example, a Gamma distribution with a very long tail might look remarkably like a Gaussian. The model correctly handles this uncertainty by assigning non-zero weights to multiple types when the data is ambiguous, rather than forcing a potentially incorrect hard classification.

  • Context Matters for Discrete Data: The paper highlights that "ordinality" isn't just about the values themselves, but the context. For example, "Duration in months" in the German dataset was identified as ordinal rather than count because its distribution was highly irregular, suggesting it behaves more like a set of ordered categories than a smooth counting process.

Key Results

  • Better Fit: On almost all real-world datasets evaluated, the proposed method achieved a significantly higher test log-likelihood compared to a baseline model that simply assumed all continuous variables were Gaussian and all discrete variables were categorical.
  • Discovery of Hidden Labels: In the "German Credit" dataset, the model correctly identified that the "Job" variable—often treated as categorical—actually behaves as an ordinal variable (ranking from unskilled to highly qualified).
  • Robustness: The MCMC algorithm proved robust to the choice of the number of latent features, showing stable performance across different model complexities.

Limitations and Future Work

While the algorithm is scalable, MCMC is inherently more computationally expensive than simple heuristics or point-estimation methods. Additionally, the current model covers six common types (Real, Positive Real, Interval, Categorical, Ordinal, Count) but does not yet handle "directional" or "circular" data (like wind direction or time of day), which the authors suggest as a future extension.

Ultimately, the authors envision this approach as a powerful "front-end" for automated machine learning (AutoML) systems, where discovered data types would automatically trigger the correct pre-processing steps, such as log-transforms or specific encoding schemes.

Summary updated

Abstract

A common practice in statistics and machine learning is to assume that the statistical data types (e.g., ordinal, categorical or real-valued) of variables, and usually also the likelihood model, is...
BibTeX
@inproceedings{valera2017automatic,
  author = {Isabel Valera and Zoubin Ghahramani},
  title = {{Automatic Discovery of the Statistical Types of Variables in a Dataset}},
  booktitle = {International Conference on Machine Learning},
  year = {2017},
  url = {https://openreview.net/forum?id=H1ZWC5bO-S},
}
Figure