Voice assistants increasingly hear compound requests such as "play some jazz and dim the living room lights". Understanding them requires more than one label per sentence. The system must find every goal (the intents), find the parameter spans (the slots, such as "jazz" or "living room"), and attach each slot to the right goal. Strong models exist for this, but they are trained centrally on pooled data. The few systems that train privately across devices assume one intent per utterance. We propose to train a small multi-intent model across devices with federated learning, meaning each device trains on its own data and shares only model updates. The model is taught one piece of grammar: which verb governs each word. A dependency parser (a tool that links each word to its grammatical head) acts as the teacher during training only. The small model learns to predict this "governing predicate" for every token, and uses that prediction to decide which intent each slot can belong to. At test time no parser runs.
The bet is specific, and it concerns how devices differ. On a phone, the data does not just have different label frequencies. Each device sees a skewed set of intent combinations: one household always pairs music with lights, another pairs calendar with messaging. A model trained this way learns shortcuts, such as "lights implies music". It then fails on combinations no single device has seen. Grammar does not change across households, and clause structure is label-free. We claim that a shared, label-free signal about predicate attachment keeps the shared encoder from absorbing these co-occurrence shortcuts. The result should be better accuracy on intent combinations never seen during training, at a model size and message size suitable for devices.
Core Claim and Relation to Prior Work
Multi-intent models already use graphs to connect tokens and intents. GL-GIN connects tokens within a sliding window and every intent to every token. Co-guiding Net passes information both ways between intent and slot label graphs. DGIF prunes token–intent edges using similarity to label descriptions. None of these graphs uses grammatical structure. The Co-guiding authors name dependency trees as future work, which motivates our proposal directly.
Syntactic sentiment analysis supplies the mechanism. RGAT runs graph attention over typed dependency edges, and its ablations single out a few relation types as carrying most of the signal. Contextual Graph Attention re-roots the dependency tree at the aspect being judged. We transfer the re-rooting idea directly: we treat each clause predicate as the anchor that slots attach to. The same line of work also constrains our design in two ways. It needs a parser at runtime, and it is fragile when the parser errs on informal text.
Encoding Syntactic Knowledge in Transformer Encoder shows how to remove the parser: one attention head is trained with a KL loss to match the parser's ancestor distribution, giving a 16M-parameter model. That work is single-intent and centralized. It also reports that dependency supervision alone slightly hurt slot F1 on SNIPS, which warns us to weight noisy parses carefully. We keep its distillation mechanism but change what is distilled. Instead of generic ancestors, the student predicts attachment to clause predicates, and that prediction gates which intents each token may vote for.
On the federated side, Federated Learning for SLU federates joint intent and slot models on single-intent ATIS and SNIPS only. FedKD trains a large local "mentor" and a small shared "mentee" that teach each other, and sends only mentee updates. We reuse FedKD's communication scheme unchanged. It has not been applied to structured prediction with interacting intents and slots. TM-FL fights device drift in federated sentiment analysis with a shared topic memory, which is a lexical, domain-bound prior. We test a structural prior in its place.
The mechanism can be stated compactly. Let $p_\theta(h \mid i)$ be the student's predicted probability that token $i$ is governed by predicate $h$. Let $q(h \mid i)$ be the parser's version, obtained by walking up the dependency tree to the nearest clause head. Let $s_{ik}$ be token $i$'s logit for intent $k$. The attachment loss and the clause-level intent score are
$$\mathcal{L}_{\text{att}} = \sum_i c_i \, \mathrm{KL}\big(q(\cdot \mid i)\,\|\,p_\theta(\cdot \mid i)\big), \qquad z_{hk} = \sum_i p_\theta(h \mid i)\, s_{ik},$$
where $c_i$ is the parser's confidence for token $i$, used to down-weight doubtful parses. Each predicate $h$ then emits its own intent from $z_{hk}$. Token-level intent voting, as in Stack-Propagation, is thereby restricted to the clause a token belongs to. Slots can only attach to intents emitted by their own clause. This replaces DGIF's similarity threshold with a learned grammatical one.
Research Program
The model has three parts. The shared mentee encoder is a small Transformer of about 16M parameters, matching the size of the syntactic Transformer above. It carries the attachment head, the token-level intent and slot heads, and a non-autoregressive slot decoder in the style of GL-GIN. Each client also keeps a larger mentor, initialized from a pretrained encoder. Parsing happens only on the device, during local training, so no text leaves the device. The mentor and mentee exchange soft intent and slot predictions as in FedKD. Only mentee updates are sent, compressed with FedKD's low-rank scheme. The attachment loss is applied to both models.
The analysis has three parts. First, we measure whether the attachment head's accuracy against held-out gold parses predicts multi-intent exact match across clients. Second, we compare how far each client's encoder drifts from the global encoder, with and without $\mathcal{L}_{\text{att}}$. Third, we check whether errors on unseen intent combinations are co-occurrence hallucinations: predicting an intent that frequently co-occurred on the training client but is absent from the utterance.
The main federated split is new, because no federated multi-intent benchmark exists in the literature we found. We partition MixATIS and MixSNIPS across clients by intent combination, using Dirichlet-skewed assignment. A fixed set of combinations is withheld from every client and appears only at test time.
The Mix benchmarks are built by joining single-intent utterances, so explicit coordination words are probably unusually strong cues. We therefore also build a stress set with three kinds of utterance: slots shared by two intents ("a flight and a hotel in Boston"), coordination without "and", and nested clauses. We build it by rewriting Mix test items and have annotators verify it. The rewriting may shift results, so we report scores on the stress set and the original test sets separately.
Baselines
The naive reference point is local-only training. Each client trains the mentee on its own data with no communication, which shows what federation buys at all. A second reference is plain FedAvg (federated averaging of client weights) on the same mentee without syntax or distillation, following Federated Learning for SLU but with a multi-intent head.
The strongest federated competitor is FedKD applied directly to a GL-GIN-style multi-intent mentee with the mentor–mentee setup of FedKD. Every component matches except $\mathcal{L}_{\text{att}}$ and clause-gated voting. This is the ablation that isolates our key ingredient. No reported result on federated multi-intent SLU was found on Lacuna, so we will run it ourselves. The reference FedKD figure is a 94.63% communication reduction relative to FedAvg on MIND, and we will check whether a similar reduction holds for structured prediction.
The accuracy ceilings are centralized models trained on pooled data. DPF reports 55.4% exact match accuracy on MixATIS and 93.1% on MixSNIPS. Co-guiding Net reports 57.5% overall accuracy on MixATIS with a RoBERTa encoder. We will rerun both on our splits, including the held-out-combination split. Their published numbers use the standard splits and do not transfer.
A third syntax variant runs a parser at test time, with RGAT-style typed graph attention added to the mentee. This measures what distillation gives up in accuracy relative to what it saves in latency.
All systems will be run under identical conditions. They share client partitions, the number of rounds, the local epochs, and a mentee parameter budget of about 16M. They receive the same hyperparameter search of twenty configurations per method, tuned on a validation split drawn from seen combinations only. We report the mean over five seeds.
Beating FedKD means two things together. The proposal must improve exact match on held-out combinations by more than the seed standard deviation. It must also not lose more than one point on seen combinations, at equal bytes communicated.
Why not just…?
Why not just fine-tune a pretrained Transformer with FedKD and let self-attention learn syntax implicitly? This is the most serious threat. Large encoders do encode a great deal of syntax. Two pieces of evidence argue against it. On sentiment, InterGCN with BERT reaches 87.12% on REST14 against 84.11% for plain BERT, so explicit structure still helped a strong encoder. In multi-intent understanding, graph interaction lifts overall accuracy well above flat models. Neither result was obtained on a 16M-parameter model under combination skew, however. The matched FedKD ablation settles the question directly. If the attachment loss adds nothing on held-out combinations, the claim is false.
Why not just split the utterance on conjunctions and run a single-intent model per piece? On the original Mix sets this heuristic may be very competitive, and we will report it as a baseline. Its failure modes are clear: slots shared across intents, coordination without "and", and nested requests, all of which Wu et al. note obscure boundaries. The stress set targets exactly these cases. If splitting matches our method there too, the proposal reduces to a heuristic and we would say so.
Why not just pretrain the syntax head centrally on public treebanks, freeze it, and federate only the task heads? This avoids on-device parsing and is cheaper. It is plausible, because grammar is shared across domains. The drift argument predicts that local attachment supervision matters because the encoder drifts during federated rounds, and a frozen syntax head cannot stop that. We test this directly with a "central-only syntax" ablation. If it matches the full method, we adopt it, since it is simpler.
Why not just run the parser at test time, as sentiment models do? Accuracy may be slightly higher. However, it adds a pipeline stage on the device, and parser errors on spoken, informal text feed directly into message passing. The runtime-parser variant above measures both the accuracy and the latency cost.
Proposed Experiments and Decisive Checks
The primary experiment compares exact match and overall accuracy on held-out intent combinations across three conditions: local-only, FedAvg, and FedKD, each with and without the attachment loss. The comparison is repeated at three levels of skew, on federated MixATIS, federated MixSNIPS, and the stress set. We also report slot F1, intent accuracy, bytes communicated per round, and on-device inference latency compared with the runtime-parser variant.
Several checks decide the question. The first is whether the gap between the syntax and no-syntax FedKD models widens as combination skew increases. That is the drift prediction, and a constant gap would suggest a generic regularizer instead. The second is whether co-occurrence hallucinations drop. The third is whether attachment accuracy correlates with per-client exact match.
The parser-noise risk seen in the syntactic Transformer work is tested by comparing confidence-weighted and unweighted $\mathcal{L}_{\text{att}}$. We also inject parse errors at controlled rates. A last transfer check reuses the same distilled attachment head on MAMS, with aspects in place of predicates, against RGAT-BERT's reported 84.52% accuracy. This tests whether the head captures general target attachment or something specific to intents.
References
- GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Filling
- Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs
- A Dynamic Graph Interactive Framework with Label-Semantic Injection for Spoken Language Understanding
- Investigating Typed Syntactic Dependencies for Targeted Sentiment Classification Using Graph Attention Neural Network
- Contextual Graph Attention Network for Aspect-Level Sentiment Classification
- Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling
- Federated Learning for Spoken Language Understanding
- Communication-efficient federated learning via knowledge distillation
- Improving Federated Learning for Aspect-based Sentiment Analysis via Topic Memories
- A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding
- Decoupling Breaks Data Barriers: A Decoupled Pre-training Framework for Multi-intent Spoken Language Understanding
- Jointly Learning Aspect-Focused and Inter-Aspect Relations with Graph Convolutional Networks for Aspect Sentiment Analysis
- A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding