A company that wants a task-oriented assistant for HR requests, banking, or colonoscopy preparation usually has a few hundred labeled conversations at most, and it cannot pool them with other organizations' logs. Such an assistant is a chatbot that completes a concrete task by tracking what the user wants. That tracking is called dialogue state tracking: keeping a running record of slot–value pairs such as hotel-area=north across a multi-turn conversation. Federated learning is the standard answer to "we can't pool the data." In federated learning, each organization trains on its own data and sends only model updates to a server that combines them. But each organization tracks a different set of slots. The HR client tracks incident-location and the travel client tracks hotel-area. There is no shared label to average over.
This proposal starts from a simple observation. Slot names differ across domains, but the decision a tracker makes about each slot at each turn does not. For any slot, the tracker decides whether the slot is untouched, carried over, newly set, set to "don't care," or set by referring back to another slot's value (for example, "a taxi to the same restaurant"). If the tracker reads each slot through its natural-language description, then the pair (dialogue context, slot description) maps to one of these five operations in every domain. That gives a label space shared by all clients even when their ontologies share nothing. We use this shared space to carry over a communication scheme so far shown only for images with shared object classes. Clients run a frozen, publicly pretrained encoder, send compact cluster centers ("prototypes") of their context–slot embeddings grouped by operation, and the server trains a small decision adapter on the pooled prototypes in one round. No gradients are exchanged.
Core Claim
The claim is that the domain-general part of dialogue state tracking can be learned federatedly from schema-conditioned operation prototypes in a single round. That part covers when a slot is updated, carried, or filled by co-reference. Learned this way, it beats both isolated per-client training and iterative weight averaging in the 1–10% data regime, with far less exposure of raw text. The gain should be largest on co-reference and update decisions, which low-resource trackers get most wrong.
Where this sits in prior work
The method comes from Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning (MPFT). MPFT freezes CLIP on each client, uploads class prototypes, and trains a global adapter on the server in one round, avoiding the "averaging trap." In that trap, weight averaging across heterogeneous domains lands in an uninformative region of parameter space. MPFT reports roughly 82% versus roughly 79% for FedAvg on DomainNet-50, and convergence in one round versus 10–50 for the baselines. MPFT assumes every client shares the label set, such as "dog" and "cat." That assumption breaks outright for dialogue ontologies, and fixing it is the contribution here.
The fix borrows from schema-guided tracking. Schema Encoding for Transferable Dialogue State Tracking (SET-DST) encodes natural-language slot and service descriptions with BERT and pretrains on Schema-Guided Dialogue so that few-shot transfer to MultiWOZ works. Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Tracking (DSGFNet) shows that co-reference, co-update, and co-occurrence relations between slots carry across domains. Both are centralized. We reuse their schema conditioning to make labels domain-free, and their relational view to define the operation set.
The motivation comes from the existing federated dialogue work, which stops short of multi-turn tracking. Federated Learning for Spoken Language Understanding federates single-utterance intent and slot parsing. Its federated-transfer setting, where clients' tasks overlap little, underperformed the other settings. That is exactly our regime, and it is a warning that naive federation across dissimilar domains can fail. Federated Natural Language Generation for Personalized Dialogue System splits shared and local parameters under FedProx, but every client solves the same open-domain generation task.
Federated Adversarial Domain Adaptation aligns features adversarially across clients and finds that disentangling style from content works much less well on text. That finding constrains our design: we disentangle through the explicit schema text rather than through a learned style/content split.
The low-resource evidence motivates the target operations. Plan, Generate and Complicate reports co-reference slot accuracy jumping from 30.35% to 64.72% at 1% data on MultiWOZ 2.4 once synthetic complex dialogues are added. Co-reference is therefore the bottleneck when data is scarce, and it is a domain-general behavior.
The privacy concern is concrete. The FILM attack (Princeton-SysML/FILM) reconstructs private text from language-model gradients at batch sizes up to 128. That threat is serious for HR data like HR-MultiWOZ or clinical dialogues like those in Bootstrapping a Conversational Guide for Colonoscopy Prep.
Research Program
The tracker. A BERT-style encoder is pretrained centrally on public Schema-Guided Dialogue following SET-DST and then frozen. For each turn and each slot, it produces a joint embedding of the dialogue history and the slot description. Three light heads sit on top of this embedding:
- an operation head that predicts one of none, carry, update, don't-care, or refer-to-slot;
- a value-match head that scores a candidate value's description against the context for categorical slots;
- a span head for free-form values.
The span head is kept frozen from public pretraining and optionally fine-tuned locally. It never leaves the client.
What clients send. Each client, holding labels only for its own domain's slots, computes embeddings for its labeled turns. It then uploads, per operation class and per match/no-match class, a small number of K-means centers. For refer-to-slot, it also uploads the embedding of the referenced slot's description, so the server can learn which slot pairs link. Because both labels are shared across every domain, prototypes from HR, hotel, and train clients land in one common, learnable space.
What the server does. The server trains the operation and match adapters on the pooled prototypes in one round. Clients then optionally distill the global adapter into a local copy using their own few labels, as in MPFT. Each client applies the Fragment Rotation augmentation from Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems to its own data before computing prototypes. That paper found that this simple non-neural method rivals LLM paraphrasing at 2–10% data, and running it locally means no seed dialogues leave the client.
Analysis. Beyond accuracy, we measure how much the pooled prototypes help each operation class separately. The prediction is that refer-to-slot and update gain most, while carry and none gain little. We also measure what an attacker can recover from prototypes, compared with what FILM recovers from FedAvg updates on the same data.
Baselines
Isolated training. The naive reference point is each client alone. A client starts from the same SGD-pretrained encoder, fine-tunes the same heads on only its own labeled dialogues, and adds local Fragment Rotation augmentation. It never communicates. This baseline tells us whether federation adds anything at all, so it uses the same encoder, heads, augmentation, and hyperparameter search budget as our method.
Iterative federated averaging. The strongest federated competitor is FedAvg and FedProx (the proximal-regularization variant used by FedNLG) applied to the full schema-conditioned tracker. Clients fine-tune the top encoder layers plus heads, exchanging updates for up to 50 rounds. Schema conditioning already gives this baseline a shared output space, so it is a genuinely fair rival rather than a straw man. It receives the same number of local epochs per round as our method's local distillation and an equal tuning budget. We report it at its best round.
Centralized upper references. We report a centralized model trained on the union of all clients' data. We also report published centralized numbers. At 1% data on MultiWOZ 2.4, EDZ-DA reports 43.82% joint goal accuracy against 37.64% for NeuralWOZ (Plan, Generate and Complicate). DSGFNet reports 56.7% joint goal accuracy on MultiWOZ 2.1 and 55.8% on MultiWOZ 2.2 with full data, and 24.4% on Schema-Guided Dialogue unseen domains (DSGFNet). For HR-MultiWOZ, no reported state-tracking result was found on Lacuna; the file reports only dataset statistics and human ratings.
What counts as beating. Our method beats the competition if it exceeds isolated training by a margin larger than the seed-to-seed spread and matches or exceeds best-round FedProx. It must do so with one round of communication and no gradient upload. Closing a substantial part of the gap to centralized training at 1% would be a strong result. Exceeding EDZ-DA's 43.82% is not expected, since EDZ-DA pools all domains centrally.
Ablations. The ablation that isolates the key ingredient is MPFT applied naively. It uses the same pipeline but keys prototypes by client-specific slot identity instead of by schema-conditioned operation, so label spaces do not overlap across clients. If this ablation performs as well as our method, the shared-operation idea is not doing the work. Two secondary ablations compare mean prototypes with K-means prototypes and remove the refer-to-slot class.
Why not just…?
Why not just pretrain on public Schema-Guided Dialogue and fine-tune locally, with no federation? This is the most serious threat. SET-DST's own results show that public schema pretraining carries much of the transfer burden, so the extra private clients may add little. The isolated baseline above tests exactly this. The proposal stands only if pooled operation prototypes improve the categories that public data covers poorly, especially co-reference in specialized domains such as HR. If the gain is negligible, we will report that as the main finding: in this regime, public schema pretraining makes cross-organization federation unnecessary.
Why not just run FedAvg on the schema-guided tracker? Once slots are described in text, averaging no longer suffers from mismatched outputs, so the averaging trap may be milder than MPFT found for images. Two pieces of evidence suggest it remains a problem. The spoken language understanding study found federated transfer weakest when tasks overlap little, and FedNLG needed parameter splitting to keep averaging from washing out client-specific behavior. Even if FedProx matches us on accuracy, it needs tens of rounds and uploads gradients, which FILM shows can leak sentences. The fair comparison above therefore reports accuracy, rounds, and measured leakage together.
Why not just prompt a large language model with the target schema? EDZ-DA's authors argue that GPT-4-class models are impractical in production because of latency, cost, and external API dependence. Sending HR or clinical dialogues to a third-party API is also exactly the data movement these organizations forbid. On quality, zero-shot schema-guided transfer reaches only 24.4% joint goal accuracy on unseen Schema-Guided Dialogue domains (DSGFNet). We will still include a few-shot prompted open model run locally as a reference on HR-MultiWOZ, so the cost of choosing a small model is visible.
Proposed Experiments
Client setup. Clients are built from MultiWOZ 2.4, using its five state-tracking domains (restaurant, hotel, attraction, train, taxi), plus selected Schema-Guided Dialogue services and HR-MultiWOZ as a sensitive client. Each client receives the dialogues that touch its domain but holds labels only for its own slots. This mimics an organization that owns one part of a conversation. We sample 1%, 5%, and 10% of each client's dialogues and use five seeds.
Evaluation. The metrics are joint goal accuracy, per-operation accuracy, and co-reference slot accuracy. In a leave-one-domain-out protocol, the held-out client contributes no prototypes and only receives the global adapter. This tests whether operation knowledge transfers to an unseen ontology. HR-MultiWOZ, with 550 dialogues over 10 domains, serves as the realistic target, evaluated both held out and as a participant.
Decisive checks. Three outcomes decide the proposal:
- Our method beats SGD-pretrained isolated training on co-reference and update accuracy at 1% data.
- The slot-identity ablation loses most of that gain.
- At equal accuracy, prototype upload yields measurably lower text recovery than FILM on FedAvg updates. Prototype recovery is measured by nearest-neighbor matching against a public candidate pool and by a trained embedding-inversion decoder.
If the first check fails, we will conclude that federation is unnecessary here rather than defend the method.
References
- Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning
- Schema Encoding for Transferable Dialogue State Tracking
- Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Tracking
- Federated Learning for Spoken Language Understanding
- Federated Natural Language Generation for Personalized Dialogue System
- Federated Adversarial Domain Adaptation
- Plan, Generate and Complicate: Improving Low-resource Dialogue State Tracking via Easy-to-Difficult Zero-shot Data Augmentation
- HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent
- Bootstrapping a Conversational Guide for Colonoscopy Prep
- Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems