Spoken language understanding systems do two linked jobs at once. They decide what the user wants (intent detection, e.g. BookFlight) and pull out the pieces of the request (slot filling, e.g. Destination = Boston). Over the past several years the field has moved away from one-way pipelines, which predict the intent first and then fill slots given that intent, toward joint models in which each task informs the other. These include cross-attention between the two tasks, label graphs, and contrastive objectives with as many as eight loss terms. Each new model reports a gain of a few points on the standard benchmarks. None of these papers reports how many training runs it took to find those points, or how much accuracy is lost when the settings move slightly. In character-level transduction, a careful study found that once both architectures received the same random-search budget, the supposedly weaker LSTM matched or beat the Transformer (Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks). Nobody has asked the same question of bidirectional intent–slot models.
We propose to answer it and to test a candidate fix. The first half is a budget-controlled re-evaluation. A one-way pipeline, a plain shared-encoder model, and three bidirectional joint models are all tuned by the same random search under the same trial budget. They are scored by the accuracy each can be expected to reach after k trials, not by its single best run. The second half tests whether a weight-tied ("Siamese") label-matching model keeps most of the benefit of multi-loss contrastive training while exposing only two tuning knobs. In a Siamese network, two inputs pass through the same encoder and are compared by distance. Here the utterance and the label names would share one encoder. The paper therefore produces two things: a cost-adjusted ranking of existing designs, and a falsifiable claim about whether tying weights shrinks the space a practitioner has to tune.
Core Claim
Our hypothesis has two parts. First, under an equal and modest tuning budget, much of the reported margin of heavily coupled joint models over a well-tuned shared-encoder baseline comes from finding a narrow good region of hyperparameters, not from a wider capacity advantage. That margin should shrink at small budgets and should transfer poorly when a configuration tuned on one dataset is reused on another. Second, mutual guidance between intent and slots can be had with very few knobs. If intent labels, slot labels and utterance tokens are all embedded by one tied encoder into one space, then intent–slot compatibility becomes a distance in that same space. It needs no extra graph or attention parameters and no per-level loss weights. The new part is the cost-adjusted comparison itself, which no paper in this area has run, together with a direct test of whether weight tying reduces tuning burden. Both parts can fail. If the bidirectional models keep their margin at every budget and keep wide equivalence regions, the first claim is wrong. If the tied model falls behind a tuned multi-head baseline, the second claim is wrong.
Where this sits in prior work
Several papers motivate the gap. A Co-Interactive Transformer for Joint Slot Filling and Intent Detection swaps queries between intent and slot representations. Co-Guiding for Multi-Intent Spoken Language Understanding passes messages between intent and slot labels over two graphs and adds supervised contrastive losses. Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning (MMCL) stacks utterance-, slot- and word-level contrastive losses with self-distillation into an eight-term objective. Its authors acknowledge that this makes tuning across domains hard. All three report point estimates after unreported sweeps. We do not propose another coupling mechanism. We measure what the existing ones cost.
One paper constrains the design. Decoupling Representation and Knowledge for Few-Shot Intent Classification and Slot Filling (JMRM) argues that tight coupling hurts few-shot transfer. It enforces intent–slot compatibility with masks at decoding time over a shared encoder and stays robust across distance functions. That result supports the bet that light coupling can be enough. It also means our tied model must beat or match JMRM-style masking, and not just beat the heavy models.
The method comes from two sources. The sensitivity metrics come from Wiemerslage et al., who studied only single-task character transduction and only uniform random search. We carry those metrics to multi-task models and add a portability test. The weight-tying idea goes back to Signature Verification Using a Siamese Time Delay Neural Network. Oreo: Detection of Clones in the Twilight Zone motivates it as both a symmetry guarantee and a regularizer, though Oreo tuned its thresholds by hand. Other work shows that complex hybrids open narrow good regions: Relational Graph Transformer, Semi-Implicit Graph Variational Auto-Encoders, and Recent Developments on ESPnet Toolkit Boosted by Conformer. These are evidence from other domains, not proof that the same happens in spoken language understanding.
Research Program
The tied label-matching model. A single pretrained encoder (BERT-base for the main comparisons, RoBERTa as a check) embeds three things: the utterance tokens, a pooled utterance vector, and the natural-language names of every intent and slot label. Intent scores are cosine similarities between the pooled utterance vector and each intent-name embedding. Slot scores are similarities between each token and each slot-name embedding. Both sets of scores are trained with temperature-scaled cross-entropy. Mutual guidance is one term computed from embeddings the model already produces. Let $\mathbf{e}_i$ be the embedding of the predicted intent and $\mathbf{e}_s$ the embedding of slot $s$. Each token's slot scores are shifted by $\beta \cos(\mathbf{e}_i,\mathbf{e}_s)$, where $\beta$ is a learned scalar. Symmetrically, the intent score is shifted by the mean compatibility of the slots predicted in the utterance. This creates two-way influence with no new attention or graph parameters.
Beyond the shared learning-rate schedule, only two hyperparameters belong to the method: the temperature, and a single weight on the slot loss. MMCL, by comparison, has seven or more loss weights plus margins. The ablation partner is identical except that the label names go through a separate, untied encoder. That isolates whether the tying itself matters.
The tuning protocol. Every model gets the same random search. The search covers learning rate, warmup, dropout, batch size and weight decay, plus each model's own knobs: loss weights, margins, cross-attention depth, graph layers. Each model gets 64 trials per dataset, and the ranges are set wide enough that the top configurations do not sit at a boundary. For each model we compute expected maximum validation accuracy as a function of trial count, by bootstrap resampling of the trials. We also compute Wiemerslage et al.'s Rel@k, the percentage of best-equivalent runs (within 1 point of overall accuracy), and the number of local maxima when runs are ordered by configuration similarity. The top five configurations are rerun with three seeds each, so that seed noise is separated from configuration effects.
Portability. The best configuration found on MixATIS is used unchanged on MixSNIPS, and the reverse. The resulting drop measures the cost of deploying a model without re-tuning, which is the cost practitioners actually pay.
Baselines
The naive reference point is a one-way pipeline: a BERT intent classifier whose predicted intent is fed as an embedding into a separate BERT slot tagger. It gets the same 64-trial search, split across its two models. The literature says this design propagates errors (A Survey on Spoken Language Understanding: Recent Advances and New Frontiers). This experiment checks how large that error propagation is once the pipeline is tuned properly.
The "do almost nothing" joint baseline is a shared BERT encoder with two linear heads and one loss weight. Under equal tuning it is the most dangerous competitor to every method in the study, including ours. We count the tied model as beating it only if its expected maximum overall accuracy is higher at every budget from 8 to 64 trials, by more than the three-seed standard deviation.
The strongest existing methods are MMCL and Co-guiding-SCL Net on the multi-intent sets and the Co-Interactive Transformer on single-intent SNIPS and ATIS, all trained from their released code where it exists. On MixATIS, MMCL reports 54.8% overall accuracy (61.6% with RoBERTa). On MixSNIPS it reports 79.5% (86.4% with RoBERTa) (Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning). The Co-Interactive Transformer with BERT reports 93.1% overall accuracy on SNIPS and 88.8% on ATIS (A Co-Interactive Transformer for Joint Slot Filling and Intent Detection).
Decoupling Breaks Data Barriers: A Decoupled Pre-training Framework for Multi-intent Spoken Language Understanding reports 55.4% exact-match accuracy on MixATIS and 93.1% on MixSNIPS. It relies on extra pretraining over 2.4 million NER examples and 136,000 single-intent SLU examples, so we cite it as context rather than as a same-data competitor. Our first check is whether our search reproduces the published numbers within about one point. If it does not, every later conclusion is qualified accordingly.
JMRM-style decode-time masking is added on top of the two-head baseline. It is the strongest light-coupling rival, and it reports 41.76% joint accuracy on 1-shot Snips (Decoupling Representation and Knowledge for Few-Shot Intent Classification and Slot Filling).
Why not just…?
Why not just auto-balance MMCL's eight losses with learned uncertainty weights or gradient normalization? This is a real threat. If automatic weighting flattens MMCL's sensitivity profile, a tied architecture is unnecessary. We therefore include MMCL with learned loss weighting as a separate arm under the same budget. The tied model is worth proposing only if it reaches comparable expected accuracy with better portability. If it does not, the paper's message becomes "auto-weight your losses," and that is still a useful finding.
Why not just tune the plain shared-encoder model harder? This may be the true answer. Wiemerslage et al. found exactly this pattern for LSTMs against Transformers. The ablations in the Co-Interactive and Co-guiding papers that remove one guidance direction were run without matched tuning, so they do not rule it out. The budget curves settle the question directly. If the two-head model sits within seed noise of everything else at every budget, we report that as the main result.
Why not just use Bayesian optimization so complex models become cheap to tune? Smarter search lowers the cost for everyone, but it does not widen a narrow good region or fix poor transfer across datasets. As a secondary arm we rerun the top two models with a Bayesian optimizer at 32 trials. If the gap between models narrows under smarter search, sensitivity was a search-efficiency problem rather than a property of the models.
Why not just use a well-tuned BiLSTM? The Co-Interactive results show pretrained encoders adding roughly three points on SNIPS (90.3% with a BiLSTM against 93.1% with BERT). So we hold the encoder fixed and vary only the coupling. The LSTM question is a different study.
Proposed Experiments and Decisive Checks
The benchmarks are MixATIS and MixSNIPS for multi-intent understanding, SNIPS and ATIS for single-intent, and a 1%-of-training-data split of MixATIS as a low-resource regime. On that split, DPF reports 37.1% accuracy while standard models barely reach 3.7%, which suggests sensitivity should be at its worst there. The metrics are overall/exact-match accuracy, intent accuracy and slot F1, together with the sensitivity measures and the portability drop described above.
Four checks decide the outcome. The first is whether the ranking of models at 8 trials differs from the ranking at 64. The second is whether bidirectional models show a smaller percentage of best-equivalent runs and larger portability drops than the two-head baseline. The third is whether the tied model beats its untied twin; if not, weight sharing is not the active ingredient. The fourth is whether the compatibility term beats decode-time masking in the 1% regime.
The cost is about 64 trials × 7 models × 5 settings, with BERT-base fine-tuning. That is affordable on a modest cluster, and we will report it alongside the results.
References
- Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
- A Co-Interactive Transformer for Joint Slot Filling and Intent Detection
- Co-Guiding for Multi-Intent Spoken Language Understanding
- Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning
- Decoupling Representation and Knowledge for Few-Shot Intent Classification and Slot Filling
- Signature Verification Using a Siamese Time Delay Neural Network
- Oreo: detection of clones in the twilight zone
- Relational Graph Transformer
- Semi-Implicit Graph Variational Auto-Encoders
- Recent Developments on Espnet Toolkit Boosted By Conformer
- A Survey on Spoken Language Understanding: Recent Advances and New Frontiers
- Decoupling Breaks Data Barriers: A Decoupled Pre-training Framework for Multi-intent Spoken Language Understanding