Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms

Tianshu Zhang, Changchang Liu, Wei-Han Lee, Yu Su, Huan Sun

Introduction

Semantic parsing—the process of translating natural language queries (e.g., "How many students are in the CS department?") into formal code like SQL—is a critical component of modern data interfaces. However, training effective neural parsers is notoriously data-intensive. While large organizations can aggregate massive datasets, many institutions like hospitals or law firms hold sensitive, private data that cannot be shared due to regulatory constraints. This creates a significant barrier for smaller institutions that lack the data volume to train high-performing models independently.

Federated Learning (FL) offers a potential solution by allowing a central server to aggregate model updates from local clients without ever accessing their raw data. However, standard FL algorithms often struggle with the extreme heterogeneity found in real-world semantic parsing tasks, where different "silos" use distinct database structures, languages, and query complexities.

Addressing Heterogeneity with Lorar

The primary contribution of this research is the introduction of Lorar (LOss Reduction Adjusted Reweighting), a novel mechanism designed to handle non-IID (not independently and identically distributed) data in cross-silo environments.

In traditional FL frameworks like FedAvg, a client's contribution to the global model is typically weighted by the size of its local dataset. The authors argue that this approach is insufficient for heterogeneous semantic parsing, as it can lead the global model to ignore the unique distributions of smaller or more complex datasets.

The Lorar mechanism adjusts client weights based on the improvement (ΔL) observed during local training rounds, ensuring that the global model adapts to diverse client distributions.

Lorar utilizes training loss reduction ($\Delta L$) as a proxy for the "distance" between the global model and a client's local optimum. If a client experiences a significant drop in loss during a local training round, it indicates that the current global model was poorly adapted to that client's specific data. Lorar assigns higher weights to these updates, allowing the global model to more aggressively incorporate the structural knowledge required by diverse clients.

Experimental Setup

To evaluate Lorar, the researchers established a realistic cross-silo FL environment using T5-base as the backbone model. They repurposed eight well-known single-domain text-to-SQL datasets to act as distinct clients, varying in size and complexity:

  • Large: Advising, ATIS
  • Medium: GeoQuery, Restaurants, Scholar
  • Small: Academic, IMDB, Yelp

This setup simulates the "messy" reality of cross-institutional collaboration, where domains range from flight bookings to academic records.

Key Results and Discussion

Lorar was tested against industry-standard FL algorithms, including FedAvg, FedOPT, and FedProx. The results demonstrate that Lorar-equipped models consistently outperform their counterparts, achieving absolute gains in Exact Match (EM) scores ranging from 4% to 20% on average.

Training loss comparison on the GeoQuery dataset shows that the Lorar-enhanced FedOPT algorithm (blue) achieves faster convergence and lower overall loss compared to standard FedOPT (red).

The most dramatic improvements were observed in small-scale clients. For example, the Academic dataset saw its accuracy jump from 2.63% to 65.79% when using Lorar. This "Small Client Effect" suggests that Lorar effectively allows smaller silos to benefit from the collective intelligence of the federation without their unique data distributions being drowned out by larger participants.

While centralized training (where all data is pooled) remains the performance ceiling, Lorar significantly closes the gap. This makes private, collaborative training a viable reality for institutions that were previously unable to leverage state-of-the-art semantic parsing due to data scarcity and privacy concerns.

Conclusion and Future Work

This work establishes a robust framework for Federated Learning in semantic parsing and introduces a reweighting mechanism that thrives on data heterogeneity. Future research will focus on quantifying the specific contributions of individual clients and exploring the application of Lorar to other code-generation tasks, such as Python or API call generation. Additionally, integrating Differential Privacy could further bolster the security of the framework against sophisticated data-leakage attacks.

Summary updated

Abstract

This paper studies a new task of federated learning (FL) for semantic parsing, where multiple clients collaboratively train one global model without sharing their semantic parsing data. By leveraging data from multiple clients, the FL paradigm can be especially beneficial for clients that have little training data to develop a data-hungry neural semantic parser on their own. We propose an evaluation setup to study this task, where we re-purpose widely-used single-domain text-to-SQL datasets as clients to form a realistic heterogeneous FL setting and collaboratively train a global model. As standard FL algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel LOss Reduction Adjusted Re-weighting (Lorar) mechanism to mitigate the performance degradation, which adjusts each client's contribution to the global model update based on its training loss reduction during each round. Our intuition is that the larger the loss reduction, the further away the current global model is from the client's local optimum, and the larger weight the client should get. By applying Lorar to three widely adopted FL algorithms (FedAvg, FedOPT and FedProx), we observe that their performance can be improved substantially on average (4%-20% absolute gain under MacroAvg) and that clients with smaller datasets enjoy larger performance gains. In addition, the global model converges faster for almost all the clients.
BibTeX
@inproceedings{zhang2023federated,
  author = {Tianshu Zhang and Changchang Liu and Wei-Han Lee and Yu Su and Huan Sun},
  title = {{Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.678},
}
Figure