How Much Word Structure Should a Noisy Channel Carry?

When text is sent over a wireless link, the system has to decide how much of the sentence's internal structure to spend channel symbols on. One option is to send only the content words, treating them as independent of one another (the "bag-of-words" or term independence assumption). The other is to also send the relations that bind those words together: who did what to whom, and which word a "not" applies to. Recent semantic communication systems for text choose a fixed answer. The TESC system extracts dependency-parsed entity–relation–entity triplets and filters them with hand-set thresholds. Neural text codecs send a fixed-length encoding of the whole sequence. In both cases the choice is a hyperparameter set by hand once, offline, and then held constant across every channel condition the system meets. We think this constant is the wrong object. When the channel is clean, relational structure is cheap to deliver and carries meaning that independent terms cannot express. When the channel is noisy, a corrupted relation symbol can attach the wrong words to each other, and that error spreads to everything built on it. Errors in independent terms stay local.

We propose to turn this hand-tuned "how much dependency to send" setting into a quantity chosen at deployment time from the measured signal-to-noise ratio (SNR). A single codec is trained once across the whole range of that setting and across the SNR range. A cheap search on held-out data then picks the setting for each channel state, so no model is retrained. The scientific claim is testable and could be false: the best amount of transmitted term dependence shifts systematically with SNR, and making it a channel-conditioned control beats both a hand-tuned constant and an automatically tuned constant.

Core Claim

Define a codec whose transmitted symbols fall into two groups. "Term" symbols carry individual content words. "Link" symbols carry dependency arcs taken from a parse, the same Spacy-style syntactic relations TESC uses. Training minimizes

$$\mathcal{L} = \mathcal{L}_{\text{task}}(\gamma) + \lambda_{\text{link}} \sum_{g \in G_{\text{link}}} \lVert z_g \rVert_2 + \lambda_{\text{term}} \sum_{g \in G_{\text{term}}} \lVert z_g \rVert_2,$$

where $\mathcal{L}_{\text{task}}(\gamma)$ is the downstream loss (classification, question answering, or reconstruction) after a simulated channel at SNR $\gamma$. Here $z_g$ is the block of channel symbols for group $g$, and $G_{\text{link}}$ and $G_{\text{term}}$ are the link and term groups. The penalties are group lasso terms, as in Linguistic Structured Sparsity in Text Categorization. That paper penalizes weights of parse-tree constituents, LDA topics, and Brown clusters together so that whole linguistic groups switch on or off. Groups driven to zero are not transmitted, so the penalty weights directly control how much structure goes on air. In that paper the penalty weights are a single global constant found by grid search. We instead feed $(\lambda_{\text{link}}, \lambda_{\text{term}})$ to the network as inputs, alongside $\gamma$, and sample them during training. One set of weights then covers the whole family of trade-offs.

The hypothesis is that the deployment-time optimum $\lambda^*_{\text{link}}(\gamma)$ rises as SNR falls. At low SNR the codec should drop toward term independence. At high SNR it should send rich structure. We also expect the slope to depend on the task: structure should matter most where meaning is compositional, such as negation in sentiment.

Why This Is Not Already Done

Three bodies of work each cover one piece and leave this question open.

The channel-adaptation literature conditions one network on channel state. Attention Feature modules in Gunduz et al. and DeepJSCC-Q rescale features from noise estimates. The Channel and Rate ModNets in SwinJSCC modulate latents from channel state and mask symbols by importance. The SNR-driven Gated Net of Lyu et al. prunes latent dimensions when the channel is clean. All of these work on images and on dense, unstructured latents. None of them asks which linguistic units to send, and their own loss weights are set by hand. We reuse their conditioning mechanism. What we change is what is being conditioned: an interpretable linguistic grouping and its penalty strength.

The text codecs supply the motivation and the closest systems. TESC shows that explicit relational triplets plus filtering can cut transmitted words by 83.8% with 0.4% performance decay. Its filter thresholds, however, are static and do not respond to channel state. Deep Learning for Joint Source-Channel Coding of Text uses a recurrent encoder to capture context but a fixed 400-bit budget. R-DeepSC recalibrates self-attention against corrupted tokens. It handles physical channel variation only by training on randomized SNRs, and it hand-balances its loss terms.

The hyperparameter optimization literature supplies the search method. TPE from Algorithms for Hyper-Parameter Optimization and Gaussian-process Bayesian optimization from Practical Bayesian Optimization of Machine Learning Algorithms beat human tuning, for example 14.98% versus 18% error on CIFAR-10. Both treat the optimum as a fixed offline point. Our change is to move the search from training time to test time. Because the codec is trained across the whole penalty range, evaluating a new $\lambda$ costs a forward pass rather than a training run. The search can therefore be repeated per channel state.

Research Program

We build one Transformer text encoder and decoder. The encoder emits term symbols aligned to content words and link symbols aligned to dependency arcs. A small conditioning network, in the style of SwinJSCC's ModNet, ingests the estimated SNR and the two penalty weights and scales feature channels in every layer. Channels are AWGN and Rayleigh fading under a fixed average power constraint. Training samples SNR uniformly over roughly −5 to 20 dB and samples the penalty weights log-uniformly over three orders of magnitude.

After training we run a two-dimensional TPE search over $(\lambda_{\text{link}}, \lambda_{\text{term}})$ on validation data at each point of an SNR grid. The objective is task accuracy at a matched average number of channel uses. We then fit a monotone map from SNR to penalty weights. To address the known fragility of adaptive codecs to channel estimation error, the controller also takes the variance of the SNR estimate as an input. We test whether it learns to lean toward term independence when the estimate is uncertain.

The analysis has three parts. First, we measure the shape of $\lambda^*(\gamma)$. Second, we measure how much of the transmitted symbol budget goes to links versus terms as SNR varies. Third, we break down errors on structure-sensitive subsets. The main structure-sensitive subset is the Stanford Sentiment Treebank's negated sentences. On that subset Recursive Deep Models for Semantic Compositionality report 71.4% for a tree model versus 33.3% for a plain recurrent net. This is direct evidence that dependency structure carries the meaning there.

Baselines

The naive reference is a codec that does nothing about structure. It is the same backbone with no link symbols at all, which is pure term independence. Alongside it we run the opposite fixed extreme, a full-sequence codec with no group penalty. These two bracket the space. If either one dominates across the whole SNR range, the hypothesis is wrong.

The hand-tuned constant baseline reproduces TESC's design point. It is our codec trained with a single fixed penalty chosen by a small manual grid at a mid-range SNR, mimicking how TESC and Yogatama and Smith set their thresholds and penalties.

The automatically tuned constant baseline gives the same codec one penalty pair selected by TPE on mixed-SNR validation data. TPE's trial budget equals the total number of evaluations our per-SNR search uses. This baseline isolates whether channel dependence helps, as opposed to merely better tuning.

The strongest existing-method competitor is an implicitly conditioned codec. It is R-DeepSC-style text transmission (R-DeepSC) with a SwinJSCC Channel ModNet added. It receives SNR as input, has no explicit link or term grouping, and its loss weights are tuned by TPE. It is the strongest competitor because it can, in principle, learn the same SNR-dependent trade-off internally without any linguistic prior.

The expensive upper reference is a set of separately trained models, one per SNR bin, each with its own TPE-tuned penalties. Total training compute is reported both matched to our single model and unrestricted.

A separate source-and-channel coding reference uses a Huffman or gzip-style compressor with an adaptive-modulation channel code. It shows the cliff effect documented in Lyu et al. and in the text JSCC work.

All learned systems share the backbone size, training data, number of channel uses, and optimizer steps. All tuned systems receive equal search budgets.

We found no reported absolute numbers on Lacuna for any text semantic-communication baseline on these benchmarks. TESC reports only relative gains: an 80.5% accuracy improvement over traditional methods on SST2 and a 150% gain on bAbI. No reported result was found on Lacuna for R-DeepSC's BLEU or BERT score values. The noise-free ceiling on binary SST is 85.4% accuracy for the tree model in Socher et al., and we use it as the high-SNR reference. We therefore re-run every baseline ourselves.

Beating the field means two things. At matched channel uses, the channel-conditioned control must exceed both constant-penalty baselines by more than seed variance in the low-SNR half of the range, and it must stay within seed variance of the per-bin oracle. It must also match or beat the implicitly conditioned codec. The key ablation freezes the penalty weights at their mixed-SNR optimum while keeping SNR conditioning in the network. This separates "conditioning on the channel" from "choosing structure by channel."

Why not just…?

Why not just feed SNR to an end-to-end Transformer and let it learn what to send? This is the real threat. The SwinJSCC ModNet already gives a single network smooth behavior across SNR on images, and a Transformer can represent dependencies without a parser. Two things in the file point to a gap but do not settle it. R-DeepSC's finding is that plain self-attention codecs degrade under token corruption unless attention is explicitly recalibrated. Yogatama and Smith found that structured groups reach equal or better accuracy with 19–27% of the features. The implicitly conditioned codec is therefore a headline baseline, not a strawman. If it matches us at matched channel uses and on the negation subset, we will report that the explicit knob buys interpretability and controllability but no accuracy. We will still report the measured SNR-to-structure curve, which is a finding in its own right.

Why not just tune the penalty once with Bayesian optimization? If the optimal penalty does not move with SNR, a single TPE-tuned constant is simpler and just as good. The decisive measurement is the per-SNR search itself. A flat $\lambda^*(\gamma)$ falsifies the core claim. A curve that is flat for AWGN but not for Rayleigh fading would narrow the claim to fading channels.

Why not just train one model per SNR bin? This approach is the accuracy ceiling. Gunduz et al. explain why it is impractical on edge devices: models must be stored and swapped, and performance drops between bins. We include it as an upper reference and claim success only if a single model closes most of the gap.

Why not just use separate compression and adaptive channel codes? The evidence in the file shows the cliff effect and poor performance under tight bit budgets for text, as reported in the text JSCC paper. We include this pipeline for completeness. It is not the comparison the claim rests on.

Proposed Experiments

We evaluate on SST2, including the negated-sentence subset, using classification accuracy. We evaluate on bAbI question answering using accuracy per task. We evaluate on English Europarl sentences using BLEU and word error rate, with the 4–30 word filtering used in the text JSCC paper. Each benchmark is run under AWGN and Rayleigh fading across the SNR grid, at three channel-use budgets, with five seeds.

The first decisive check is the shape of $\lambda^*_{\text{link}}(\gamma)$. We predict it decreases monotonically with SNR. We also predict it is steeper on bAbI and on SST2's negation subset than on Europarl reconstruction, where local term content dominates.

The second check is whether the low-SNR gain over the constant-penalty baselines survives against the implicitly conditioned codec.

The third check concerns channel estimation error. We inject error into the SNR estimate at deployment, without retraining, and test whether the variance-aware controller degrades more gracefully than both the constant-penalty and the implicitly conditioned codecs.

A clean positive result would show that term independence is not simply a crude approximation. It would show that independence is the right encoding when the channel is bad, and that the hyperparameter controlling it belongs to the deployment loop rather than the training script.