Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information

Esther Rolf, Michael I. Jordan, Benjamin Recht

Introduction

Machine learning datasets frequently include "side information"—structural indices like timestamps, geographic coordinates, or frame numbers. While these variables are highly informative, they present a significant challenge: using them as direct features often leads to overfitting. A model might memorize specific coordinates or times rather than learning generalizable patterns, resulting in poor performance on unseen regions or future time steps. Traditional solutions like semi-supervised learning (SSL) or spatial regularization are often computationally expensive and difficult to integrate with non-differentiable models like Random Forests.

This paper introduces Post-Estimation Smoothing (P-ES), a model-agnostic post-processing technique that leverages structural indices to improve predictions without the risks of direct feature inclusion or the overhead of complex retraining.

The P-ES Framework

P-ES decouples feature-based prediction from structural smoothing. The process involves two distinct steps:

  1. Base Prediction: An off-the-shelf model $f(x)$ is trained on the primary features to generate initial predictions $\hat{y}$.
  2. Smoothing: A linear smoothing operator $S(t)$, based solely on the structural index $t$, is applied to the initial predictions: $\tilde{y} = S(t)\hat{y}$.

The smoothing matrix $S_c(t)$ is defined as a convex combination of a local weight matrix $W(t)$ (e.g., a Gaussian kernel) and the identity matrix $I$: $$S_c(t) = c \cdot W(t) + (1-c) \cdot I$$

The parameter $c$ controls the degree of smoothing. Theoretically, the authors demonstrate that this operation reduces the expected Mean Squared Error (MSE) by averaging out uncorrelated prediction errors while preserving the underlying signal present in the structural index.

P-ES reduces prediction error in simulated settings by smoothing over structural indices.

Experimental Results

The authors evaluated P-ES across diverse domains, including simulated data, human pose estimation in video, and large-scale real estate pricing.

Human Pose Estimation

On the 3DPW dataset, applying P-ES to a per-frame baseline improved key-point accuracy (PCK) and significantly reduced acceleration error—a measure of temporal jitter—from 37.41 to 20.46. Notably, P-ES even enhanced state-of-the-art models that already incorporated temporal dynamics, suggesting it captures residual structural information.

House Price Prediction

Using the Zillow ZTRAX dataset (~600,000 home sales), P-ES boosted the $R^2$ of an XGBoost model from 0.458 to 0.526. Compared to complex semi-supervised methods like Gaussian Harmonic Energy Minimization (HEM), P-ES achieved comparable accuracy while being approximately 30x faster, adding negligible compute time (27 seconds vs. HEM's ~900 seconds).

Robustness to Distribution Shift

A key advantage of P-ES is its robustness to distribution shifts. In spatial extrapolation tests (training on Northern US data and testing on Southern US data), models that included latitude and longitude as features suffered catastrophic performance drops. In contrast, P-ES applied to a location-agnostic model successfully incorporated spatial context without degrading out-of-sample performance.

P-ES maintains performance under spatial distribution shifts where models using indices as features fail.

Conclusion

Post-Estimation Smoothing offers a simple, fast, and effective baseline for incorporating side information into machine learning pipelines. By decoupling prediction from smoothing, it provides a "plug-and-play" solution that improves accuracy and robustness across various models and tasks. While its success depends on the quality of the base model, its efficiency and ease of implementation make it a powerful tool for practitioners dealing with structured observational data.

Summary updated

Abstract

Observational data are often accompanied by natural structural indices, such as time stamps or geographic locations, which are meaningful to prediction tasks but are often discarded. We leverage se...
BibTeX
@inproceedings{rolf2020post,
  author = {Esther Rolf and Michael I. Jordan and Benjamin Recht},
  title = {{Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
}
Figure