Factor Graphs for Relational Regression

Sumit Chopra2009

Introduction

In traditional machine learning, the Independent and Identically Distributed (I.I.D.) assumption serves as a foundational simplification: we treat each data point as an isolated entity. However, real-world phenomena—from real estate markets to social networks—are inherently relational. A property's value is not merely a function of its square footage but is deeply intertwined with the values of neighboring properties and the temporal trends of its local market.

In the dissertation Factor Graphs for Relational Regression, Sumit Prakash Chopra, supervised by Yann LeCun, proposes a framework to move beyond "flat" models. By leveraging Energy-Based Models (EBMs) and Relational Factor Graphs, this work addresses the complexities of spatio-temporal dependencies in continuous regression tasks.

The Challenge of Relational Regression

While Statistical Relational Learning (SRL) has seen significant success in classification tasks (e.g., labeling the topic of a webpage based on its hyperlinks), regression in relational contexts remains challenging. In domains like real estate, the most critical predictive features are often latent. For instance, "neighborhood desirability" is a powerful driver of price, yet it is rarely captured as a discrete variable in a spreadsheet. Instead, it exists as a latent manifold reflected in the prices of nearby, contemporaneous sales.

Standard regression models ignore this structure, treating every transaction as an independent event. Chopra’s research aims to explicitly model this hidden manifold, allowing the system to learn the underlying spatio-temporal context that governs continuous variables.

The Relational Factor Graph Framework

The core of this approach is the Relational Factor Graph, built upon the principles of Energy-Based Models. Unlike probabilistic models that require the calculation of complex partition functions (integrals that are often computationally intractable), EBMs assign a scalar "energy" to different configurations of variables. In this framework, lower energy corresponds to higher compatibility between variables.

Factor Types

The model utilizes two distinct types of factors to define the energy surface:

  1. Non-Relational Factors: These capture the relationship between a specific entity’s intrinsic features (e.g., a house's number of bedrooms or lot size) and its target value (price).
  2. Relational Factors: These link multiple data points, enforcing "smoothness" constraints. They operate on the assumption that entities close to one another in space and time should share similar latent characteristics, such as desirability.

Collective Inference and Learning

Because the data points are interconnected, the model must perform collective inference. Rather than predicting prices in isolation, the system adjusts the latent variables for the entire dataset simultaneously to find the global minimum energy state.

Learning is achieved through a two-phase, EM-like procedure:

  • Inference Phase: With model parameters fixed, the system solves for the hidden desirability variables. Chopra demonstrates that if the energy functions are quadratic, this complex relational inference can be collapsed into a single, efficient calculation (Lemma 3.1), making it feasible to process millions of records.
  • Learning Phase: With the hidden variables fixed, the model updates its parameters—such as the weights of an underlying neural network—using stochastic gradient descent.

Empirical Evaluation: The Los Angeles Housing Market

The framework was tested using a massive dataset from First America Corelogic, covering approximately 1.3 million transactions in Los Angeles County over a 24-year period (1984–2008). The data included transaction prices, dates, GPS coordinates, and property attributes.

Key Results

The Relational Factor Graph significantly outperformed standard benchmarks:

  • Accuracy: The relational model placed 39.5% of houses within a 5% error margin of their actual sale price. In contrast, standard linear regression achieved only 26.6%, and a standard fully connected neural network reached 33.8%.
  • Index Robustness: When used to construct a House Price Index (HPI), the model proved more robust than the industry-standard Case-Shiller index. While Case-Shiller often exhibits systematic bias during market downturns or with extreme property values, the relational model absorbed these fluctuations into the spatio-temporal latent surface, eliminating superfluous patterns.

Insights and Future Directions

Chopra’s work demonstrates that the "desirability manifold" is not a static map but an evolving surface that can be learned from data. Furthermore, by allowing for non-linear architectures within the factors, the framework provides a flexibility that previous log-linear relational models lacked.

However, the research also highlights significant computational hurdles. Managing the neighborhood matrices for 1.3 million transactions required substantial disk space and memory, suggesting that national-scale implementation would require advanced distributed computing. Future work may also explore dynamic neighborhoods—learning the structure of relationships themselves rather than relying on fixed spatial distances.

By treating data as a community of related observations rather than a collection of strangers, this research provides a robust blueprint for the future of regression in interconnected datasets.

Summary updated

BibTeX
@misc{chopra2009factor,
  author = {Sumit Chopra},
  title = {{Factor Graphs for Relational Regression}},
  year = {2009},
}
Figure