Referee-Meta-Learning for Fast Adaptation of Locational Fairness

Weiye Chen, Yiqun Xie, Xiaowei Jia, Erhu He, Han Bao, Bang An, Xun Zhou

Abstract

Machine learning models often exhibit significant performance disparities across geographic locations, a phenomenon known as locational bias. This bias can lead to inequitable outcomes in critical applications such as agricultural subsidy distribution or traffic safety management. We introduce Referee-Meta-Learning (Meta-Ref), a framework that addresses this challenge by treating fairness as a dynamic optimization problem. Meta-Ref employs a "referee" network to assign location-specific learning rates, enabling models to adapt rapidly to new regions while minimizing performance gaps between local patches. Our experiments on satellite crop classification and traffic risk estimation demonstrate that Meta-Ref consistently outperforms existing meta-learning and domain adaptation methods in both accuracy and locational fairness.

Introduction: The Challenge of Locational Fairness

While fairness in machine learning is often discussed in the context of demographic attributes like race or gender, locational fairness is equally critical for equitable resource distribution. A model used for satellite-based crop monitoring might perform well in one county but fail in another due to differences in soil, climate, or farming practices.

The primary technical challenge is the "non-stationary" nature of geographic data. Unlike static categories, locational groups shift; a model trained on a specific set of cities must be able to generalize to entirely new regions. Standard fairness techniques often struggle with this spatial shift. The goal is to achieve fast adaptation (few-shot learning) in a new region while ensuring that prediction quality remains consistent across all local areas within that region.

Spatial task sampling and data types for training and testing locations.

Methods: The Meta-Ref Framework

Meta-Ref builds upon Model-Agnostic Meta-Learning (MAML) by introducing a specialized "Referee" network. This referee acts as a balancer, learning how to guide the predictor model toward fair outcomes.

Three-Phase Training Process

The framework operates in three distinct phases:

  1. Performance Estimation: The model calculates local prediction losses and performance metrics across a cluster of locations (a "spatial task").
  2. Fairness-Aware Learning Rate Estimation: The Meta-Ref network analyzes local performance relative to the global average, alongside data features. It then assigns a specific learning rate to each location. Locations where the model struggles receive higher learning rates, forcing the model to prioritize them.
  3. Dual Meta-Updates: The system performs a "gradient of gradients" update. It simultaneously updates the prediction model for accuracy and the Meta-Ref network to ensure its learning rate assignments effectively improve fairness in subsequent rounds.
The Meta-Ref architecture, illustrating the interaction between the Prediction model and the Meta-Ref referee during the meta-learning process.

Key Innovations

  • Relative Performance Invariance: By focusing on the difference between local error and global average, the referee remains effective even as the overall model improves, allowing for better transferability to new regions.
  • Sigmoid Bounding: To stabilize early training, a sigmoid-shaped curve gradually expands the range of possible learning rates as the model matures.

Experimental Evaluation

The researchers evaluated Meta-Ref on two high-impact real-world scenarios:

  • Satellite-based Crop Classification: Mapping walnut plantations in California’s Central Valley using Sentinel-2 multispectral data.
  • Traffic Accident Risk Estimation: Predicting daily accident counts in Iowa based on weather, road conditions, and historical traffic records.

To measure success, the authors utilized the Adjusted Locational Fairness (ALF) metric, which penalizes models that achieve low variance by simply being equally inaccurate across all locations.

Results and Discussion

Meta-Ref consistently outperformed standard Deep Neural Networks (DNN), LSTMs, and standard MAML in both accuracy and fairness:

  • Crop Monitoring: Meta-Ref achieved the highest F1-score (~0.71) and the lowest Locational Fairness (LF) score (~0.107), indicating the most consistent performance across counties.
  • Traffic Safety: In the regression task, Meta-Ref achieved the lowest RMSE (0.131) and the best fairness scores (0.102 LF), demonstrating superior adaptation to complex spatial shifts.

Conclusion

Meta-Ref provides a robust solution for the problem of locational bias in spatial data. By framing fairness as a meta-learning task guided by a "referee," the model achieves both high accuracy and equitable performance across diverse geographic regions. Future work will focus on incorporating policy-defined fairness criteria and reducing the reliance on labeled data for few-shot adaptation.

Summary updated

Abstract

When dealing with data from distinct locations, machine learning algorithms tend to demonstrate an implicit preference of some locations over the others, which constitutes biases that sabotage the spatial fairness of the algorithm. This unfairness can easily introduce biases in subsequent decision-making given broad adoptions of learning-based solutions in practice. However, locational biases in AI are largely understudied. To mitigate biases over locations, we propose a locational meta-referee (Meta-Ref) to oversee the few-shot meta-training and meta-testing of a deep neural network. Meta-Ref dynamically adjusts the learning rates for training samples of given locations to advocate a fair performance across locations, through an explicit consideration of locational biases and the characteristics of input data. We present a three-phase training framework to learn both a meta-learning-based predictor and an integrated Meta-Ref that governs the fairness of the model. Once trained with a distribution of spatial tasks, Meta-Ref is applied to samples from new spatial tasks (i.e., regions outside the training area) to promote fairness during the fine-tune step. We carried out experiments with two case studies on crop monitoring and transportation safety, which show Meta-Ref can improve locational fairness while keeping the overall prediction quality at a similar level.
BibTeX
@inproceedings{chen2024referee,
  author = {Weiye Chen and Yiqun Xie and Xiaowei Jia and Erhu He and Han Bao and Bang An and Xun Zhou},
  title = {{Referee-Meta-Learning for Fast Adaptation of Locational Fairness}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30197},
}
Figure