GeoAggregator: An Efficient Transformer Model for Geo-Spatial Tabular Data

Rui Deng, Ziqi Li, Mingshu WangAAAI 2025 · DOI

TL;DR

This paper introduces GeoAggregator, a novel transformer-based model designed to efficiently handle geospatial tabular data. By incorporating geographical priors through Gaussian-biased attention and a Cartesian product mechanism, GeoAggregator achieves state-of-the-art performance on various spatial regression tasks while significantly reducing computational complexity compared to existing deep learning methods.

What Problem Does This Paper Address?

Geospatial data, increasingly abundant due to location-based services, presents unique modeling challenges. Traditional statistical and geo-statistical models often rely on strong assumptions and struggle with scalability and flexibility for large, complex datasets. While deep learning, particularly graph-based networks and convolutional approaches, has shown promise, these methods often suffer from large model sizes and high computational costs, limiting their application to massive datasets. Existing transformer models, while effective for some geospatial tasks, also face quadratic complexity with input sequence length, making them impractical for large-scale geospatial tabular data. This paper aims to bridge this gap by developing an efficient and scalable transformer model that explicitly accounts for spatial autocorrelation and heterogeneity in tabular data.

Methods

GeoAggregator employs an encoder-processor-decoder architecture. Its core innovations lie in:

  • Gaussian-Biased Local Attention: Unlike standard attention mechanisms, GeoAggregator incorporates a Gaussian bias derived from spatial proximity. This bias helps the model focus on relevant neighboring points, explicitly modeling spatial autocorrelation.
  • Multi-head Cartesian Product Attention (MCPA): To manage computational complexity and model size, MCPA is introduced. It leverages the Cartesian product of embeddings from covariates and target variables, effectively enriching feature interactions without a quadratic increase in parameters. This is further enhanced by using "inducing points" in the processor modules, which reduce the complexity from quadratic to linear with respect to the input sequence length.
  • 2D Rotary Positional Embedding: To inject spatial location information, a novel rotary positional embedding is used, which augments the embedding features with 2D spatial location information through rotation matrices.
  • Geographical Priors: The model explicitly incorporates geographical priors, such as spatial autocorrelation (SA) and spatial heterogeneity (SH), which are often overlooked by standard deep learning models.

Datasets

The paper evaluates GeoAggregator on a combination of synthetic and real-world datasets:

  • Synthetic Datasets: Eight synthetic datasets were generated using four spatial processes (Linear, Spatial Lagged, Spatial Lagged X, and Spatial Durbin Models) with two types of covariates (uniform and DEM-derived). These datasets are designed to test the model's ability to handle different spatial effects.
  • Real-World Datasets:
    • PM25: A dataset of PM2.5 concentrations across mainland China, characterized by long-range spatial regression and sparse data.
  • Housing: Housing prices in King County, WA, USA, representing a small-scale, densely distributed dataset with significant spatial effects.
  • Poverty: Socioeconomic variables estimating poverty levels across the continental US, presenting a mid-range spatial regression challenge with densely distributed neighborhood data.

Key Insights and Ideas

  • The Power of Geographical Priors: Explicitly incorporating spatial autocorrelation and heterogeneity through mechanisms like Gaussian bias significantly improves performance, demonstrating that domain knowledge is crucial for geospatial modeling.
  • Scalability through MCPA and Inducing Points: The MCPA mechanism, combined with inducing points, is a key enabler for handling large geospatial datasets. It allows transformer architectures to scale linearly with input size, a major breakthrough for this domain.
  • Balancing Attention Bias: The effectiveness of the Gaussian bias is sensitive to its magnitude. Too little or too much bias can degrade performance, highlighting the importance of learning or carefully tuning this parameter.
  • Robustness to Neighborhood Size: GeoAggregator's ability to efficiently aggregate information from local sequences, even with large neighborhoods, makes it robust to less relevant data points, a common issue in other models.

Key Results

  • Synthetic Datasets: GeoAggregator models achieved the best or second-best performance on nearly all synthetic datasets, outperforming other transformer-based and convolutional models.
  • Real-World Datasets: GeoAggregator models achieved two "best" results on the real-world datasets, demonstrating strong performance on complex, real-world spatial regression tasks. For instance, on the Housing price dataset, GA-mini models achieved state-of-the-art results.
  • Computational Efficiency: GeoAggregator models, especially those using MCPA, exhibited significantly lower #Param and #FLOPs compared to vanilla attention models and other deep learning baselines, making them highly scalable and efficient. For example, GA-mini had only 4.3K parameters and 0.7M FLOPs on the Housing dataset, a fraction of what other deep learning models required.

Limitations and Future Work

  • Limitations:
    • The paper notes that for the PM25 dataset, GeoAggregator models showed less satisfactory results than the best-performing model, possibly due to the long-range dependence contributing less to the target point and the complex spatial heterogeneity.
  • The performance of GeoAggregator can be sensitive to the choice of the attention bias factor ($\lambda$), requiring careful tuning or learning.
  • Future Work:
    • Tailoring the bias for each spatial feature individually could further improve performance.
  • Incorporating new input heads to handle categorical variables would enhance the model's versatility.
  • Further exploration of hyperparameter tuning (e.g., $l_{hidden}$, $L$, $l_{max}$) could lead to even better results.

Summary updated

BibTeX
@inproceedings{deng2025geoaggregator,
  author = {Rui Deng and Ziqi Li and Mingshu Wang},
  title = {{GeoAggregator: An Efficient Transformer Model for Geo-Spatial Tabular Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33259},
}
Figure