Distance Based Modeling of Interactions in Structured Regression

Ivan Stojkovic, Vladisav Jelisavcic, Veljko Milutinovic, Zoran ObradovicIJCAI 2016

Introduction

Structured regression involves predicting multiple related continuous variables simultaneously, such as temperatures across a sensor network or admission rates in various hospitals. Traditional models, like Gaussian Conditional Random Fields (GCRF), typically rely on a "smoothing" assumption: related variables are expected to have similar values. In these models, interaction potentials penalize differences between neighboring nodes, effectively pulling them toward equality.

However, real-world data often violates this assumption. Related variables may maintain consistent offsets or exhibit complex, non-identical relationships. Forcing similarity in such cases introduces systematic error. This paper introduces Distance-based Gaussian Conditional Random Fields (DGCRF), a framework that allows models to learn and enforce specific "distances" between variables, moving beyond simple smoothing.

Methodology: From Rubber Bands to Springs

The core innovation of DGCRF lies in its treatment of interaction potentials. While a standard GCRF acts like a rubber band—only exerting force to pull variables together—the DGCRF acts like a spring with a learned "nominal length."

The Distance Term

The authors introduce a distance function, $D_{ij}(X)$, into the interaction potential. This term represents the expected difference between node $i$ and node $j$ as a function of the input features $X$. The model exerts force if the actual difference deviates from this expected distance, whether the variables are too far apart or too close together.

Learning and Inference

DGCRF maintains the multivariate Gaussian form of the original GCRF, ensuring that inference (finding the most likely values) remains an efficient algebraic calculation. The distance functions $D_{ij}(X)$ are pre-trained using standard regression techniques, such as neural networks, to estimate differences between connected nodes. The overall model parameters are then optimized via maximum likelihood.

The Intuition: Filtering Correlated Noise

A key advantage of modeling differences directly is the ability to filter out correlated noise. In many systems, individual variables are highly noisy, but because they share common noise sources, their relative differences are much more stable.

Illustration of noise filtering: while individual responses $y_1$ and $y_2$ are noisy, their difference $\Delta y_{12}$ reveals a clearer underlying pattern.

As shown in Figure 1, while the individual signals $y_1$ and $y_2$ might appear chaotic, their difference $\Delta y_{12}$ often follows a much cleaner, more predictable trajectory. By capturing this relationship, DGCRF can achieve higher precision than models that treat each variable independently or force them to be identical.

Experimental Evaluation

The authors evaluated DGCRF against unstructured Neural Networks (NN) and standard GCRFs across several domains:

  1. Synthetic Tasks: Linear Chain and 2D Grid structures designed to test the recovery of known interaction patterns.
  2. Healthcare (SEPSIS): Predicting monthly sepsis admission rates across 231 California hospitals.
  3. Climate (RAIN): Estimating monthly precipitation at 1,132 US locations using meteorological features like air pressure and humidity.

Key Results

DGCRF consistently outperformed the baselines. On synthetic tasks, the model demonstrated a significant reduction in Root Mean Square Error (RMSE).

RMSE comparison across synthetic tasks, showing DGCRF's superior performance over standard GCRF and unstructured neural networks.

In real-world applications, the improvements were equally notable. For the RAIN dataset, DGCRF achieved an RMSE of 1.767 (compared to 1.790 for GCRF). In the SEPSIS dataset, it reached an RMSE of 1.238 (compared to 1.265 for GCRF). These results were statistically significant, confirming that distance-based modeling provides a robust advantage in complex, structured environments.

Conclusion

By relaxing the traditional smoothing constraint, Distance-based Gaussian Conditional Random Fields offer a more flexible and accurate approach to structured regression. The model successfully captures inherent offsets and filters correlated noise while maintaining the computational efficiency required for large-scale applications. Future work may explore the application of distance-based potentials to other graphical model architectures and methods for learning the underlying graph structure automatically.

Summary updated

BibTeX
@inproceedings{stojkovic2016distance,
  author = {Ivan Stojkovic and Vladisav Jelisavcic and Veljko Milutinovic and Zoran Obradovic},
  title = {{Distance Based Modeling of Interactions in Structured Regression}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure