Stopping When the Stop Signal Itself Is Noisy: Belief-State Control for Learned Variable-Length Feedback Codes

A feedback code is a communication scheme in which the receiver continuously reports back what it has heard, so the transmitter can adapt what it sends next. A variable-length feedback code goes one step further. It keeps transmitting only until the receiver is confident, which lets it spend fewer channel uses on easy noise realizations and more on hard ones. The best learned scheme of this kind, DeepVLF, decides when to stop sending each small group of message bits by comparing the receiver's confidence to a fixed threshold. It then tells the transmitter which groups are done over a separate control link that it assumes is perfect and free. On the standard benchmark this reaches a block error rate below $10^{-5}$ at 1 dB signal-to-noise ratio and rate 1/2 (Variable-Length Feedback Codes via Deep Learning). Real feedback links are noisy and cost channel uses. Once the "stop" signal can be corrupted, the transmitter no longer knows what the receiver has decoded. The two ends can then fall out of sync: the transmitter goes silent on a group the receiver has not finished, or keeps sending on one it has.

We propose to treat stopping, and the feedback that announces it, as decisions in a partially observable Markov decision process (POMDP). An ordinary Markov decision process describes an agent that sees the full state of its environment and picks actions to maximize long-run reward. A POMDP describes an agent that only receives noisy clues about that state. Here, the transmitter keeps a learned estimate of what the receiver currently believes, and the receiver chooses what feedback to send given what it expects the transmitter to infer. Both are trained against a reward that charges for every forward channel use, every feedback channel use spent on control, and every decoding error. The claim is specific: when control signaling must travel over the same noisy feedback link and be paid for, a belief-state stopping policy should beat threshold stopping with a separately protected control code. It should do so on block error rate at matched total rate, with the largest gains appearing as feedback quality degrades. On a perfect control channel we expect little or no gain, and the experiments are designed to confirm that too.

Why this and not the obvious MDP recast

The obvious idea at this intersection is to recast DeepVLF's stopping rule as an optimal-stopping MDP on an AWGN channel (AWGN means additive white Gaussian noise, a memoryless channel) with perfect feedback. We think this would mostly rediscover a well-tuned threshold. With a noiseless control link and a memoryless channel, the receiver's posterior over each bit group is a nearly sufficient summary of the state. A threshold that varies with the round index approximates the optimal stopping rule for this kind of problem well. A skeptical reviewer would propose exactly that per-round threshold, and would probably be right.

The regime where a sequential-decision formulation earns its cost is the one both existing bridges leave open. On the error exponent of variable-length block-coding schemes over finite-state Markov channels with feedback casts variable-length coding over channels with memory as a stochastic control problem and derives the error exponent. It assumes, however, that the channel state is seen noiselessly at both ends, and it gives no finite-length algorithm. DeepVLF provides a finite-length algorithm but assumes an error-free control link. Our proposal removes both assumptions at once, in the finite-blocklength setting. The tension noted in the literature is that asymptotic control analyses ignore short-block penalties, while short-block learned codes lack a decision-theoretic treatment of stopping. We address that tension by optimizing the finite-horizon reward directly rather than a steady-state average.

Core Claim

Let the message be split into $G$ bit groups. Let $b_t$ be the receiver's posterior over the groups after round $t$, and let $\hat b_t$ be the transmitter's estimate of $b_t$, computed only from the noisy feedback it has received. The transmitter acts on $\hat b_t$. It decides which groups to keep sending and how to split power among them. The receiver acts on $b_t$. It decides what to feed back, choosing among raw channel outputs, a coded stop mask, or a mixture, within a fixed per-round feedback budget. Both policies are trained to maximize

$$ J = -\,\mathbb{E}\big[\tau_f + \beta\,\tau_b + \lambda\,\mathbf{1}\{\hat m \neq m\}\big], $$

where $\tau_f$ and $\tau_b$ are the forward and feedback channel uses spent before the session ends, $\beta$ is the price of one feedback use relative to a forward use, $\lambda$ is the price of a block error, and $\hat m \neq m$ means the decoded message differs from the sent one. Sweeping $\lambda$ traces out the curve of error rate against rate. Because control bits are charged through $\tau_b$ and travel over the noisy link, the policy has to learn how much to protect its stop decisions and when it is safe to stop without confirmation. A fixed threshold has no way to make these trade-offs.

Research Program

We would build on the DeepVLF architecture, keeping its encoder, decoder, and variable-depth feature extractor, and replace three of its pieces. First, the perfect control link is removed, and the stop mask must be carried inside the noisy feedback symbols. Second, the transmitter gains a small recurrent network that outputs $\hat b_t$. It is trained with a cross-entropy loss against the true receiver posterior, which is available in simulation. Third, the per-group decisions to stop or continue at both ends become stochastic policies with $\hat b_t$ or $b_t$ as input, together with the round index and any available channel-state estimate. These policies are trained by policy gradient on $J$.

We borrow two ideas from other work. From Breiman meets Bellman: Non-Greedy Decision Trees with MDPs, which makes decision-tree construction tractable by letting the greedy method propose candidate actions and then optimizing over them, we take the idea of using the threshold rule as a warm start. The policy is initialized to imitate DeepVLF's threshold and then improved, which avoids learning from scratch in a very large action space. Because the reward arrives mostly at the end of the session, we expect credit assignment across rounds to be hard (Temporal Regularization for Markov Decision Process documents this difficulty). We will therefore compare shaping the reward with the per-round channel-use cost against a terminal-only reward.

The second testbed is a channel with memory, a two-state Gilbert–Elliott fading channel whose state is hidden from both ends. This is the setting where On the error exponent of variable-length block-coding schemes over finite-state Markov channels with feedback shows that inputs play a "dual role": they carry information and also steer future channel conditions. Here the transmitter's belief must also track the channel state. This tests whether the POMDP view helps beyond feedback noise alone. We deliberately do not attempt a scheme that actively steers the channel state. On fading channels that the transmitter cannot influence, such steering does not apply, so we test only inference of the hidden state.

Baselines

The naive reference point is a fixed-length learned feedback code: GBAF (All you need is feedback: Communication with block attention feedback codes), a transformer-based feedback code that always runs a set number of rounds. It is trained on the same message length $K=51$ and the same forward and feedback signal-to-noise ratios, with the round count chosen to match average rate. Under perfect feedback at 1 dB and rate 1/2, DeepVLF reports that GBAF reaches a block error rate of roughly $10^{-2}$. We also report a feedback-free reference: a fixed-length short-block code evaluated against Efficient Error-Correcting Codes in the Short Blocklength Regime, which reports that the best (128, 64) codes operate within 0.1 to 0.5 dB of the random-coding-union bound at a block error rate of $10^{-4}$.

The strongest existing method is DeepVLF, which reports a block error rate below $10^{-5}$ at 1 dB forward signal-to-noise ratio and rate 1/2 with perfect control. It also reports that DeepVLF "remains superior" with 20 dB feedback noise, but the file gives no specific error rate for that case. To compare fairly, we give DeepVLF what the proposal must itself pay for. Its stop mask is sent over the same noisy feedback link, protected by a repetition or short block code whose strength is tuned, and those uses are counted in $\tau_b$. Its threshold is tuned per round by grid search under the same simulation budget we give policy training. We call this tuned DeepVLF with coded mask. It is the baseline that matters. Beating it means a lower block error rate at equal total rate $K/(\tau_f+\beta\tau_b)$, or equal error rate at higher total rate. The margin must exceed the confidence interval from at least $10^7$ test blocks at the $10^{-5}$ operating point.

Three ablations isolate what is new. The first removes the transmitter belief network and gives the policy only raw feedback, which tests whether the belief estimate matters. The second trains the same policy with a myopic reward that looks only one round ahead, which tests whether planning across rounds matters. The third keeps a perfect control link, which should shrink our advantage toward zero if the claim is right. Finally, a Yamamoto–Itoh-style confirmation phase, in which the receiver sends a single acknowledge-or-not bit at a fixed checkpoint, is included as the structured alternative from Variable-Length Feedback Codes over Known and Unknown Channels with Non-vanishing Error Probabilities. That paper reports its performance only as bounds, not simulations, and the file contains no simulated result for such a scheme on this benchmark.

Why not just…?

Why not just tune DeepVLF's threshold per round? On an AWGN channel with perfect control, this probably matches any MDP policy, which is why we do not make that claim. The per-round threshold is included inside our strongest baseline. The proposal predicts that a threshold fails specifically when the transmitter's view of the receiver is uncertain. In that case the right decision depends on how uncertain the transmitter is, and a function of the receiver's confidence alone cannot express that. If tuned thresholds match our policy under 10 to 20 dB feedback noise, the claim is refuted.

Why not just protect the stop mask with a strong code? This is the most serious threat. A heavily protected mask removes desynchronization but costs feedback uses in every round. The learned policy's possible advantage is to spend protection only when it is needed, for example on rounds and groups where a premature stop would be costly. The coded-mask baseline, with its code strength swept, measures this directly. If the best fixed protection level matches us, the gain is only a better choice of that level, and we will report it that way.

Why not just use a Yamamoto–Itoh confirmation phase? One confirmation bit at the end is cheap and comes with second-order guarantees in the analysis of Variable-Length Feedback Codes over Known and Unknown Channels with Non-vanishing Error Probabilities. However, it confirms the whole message at once, whereas DeepVLF's per-group stopping appears to drive much of its reported gain over GBAF. Simulating the confirmation scheme at $K=51$ shows how much of the gain survives without per-group control.

Why not just use fixed-length 5G-style codes and skip feedback? Efficient Error-Correcting Codes in the Short Blocklength Regime shows these codes sit close to the finite-length bound. Our comparison therefore counts feedback channel uses at price $\beta$. If a feedback code only wins when $\beta=0$, that is not a practical win, and we report results as a function of $\beta$ to show where the crossover lies.

Why not solve the POMDP exactly? The state includes a posterior over $2^3$ values for each of 17 groups, plus hidden channel state. That is far too large for exact belief-space solvers, and robust formulations such as Robust partially observable Markov decision process are harder still. A learned belief network is the tractable approximation. We check its quality by measuring how far $\hat b_t$ is from $b_t$.

Proposed Experiments

All experiments use $K=51$ bits in groups of 3, matching DeepVLF. The first run is a sanity check. With perfect control on AWGN at 1 dB, the policy should recover DeepVLF's reported error rate below $10^{-5}$ at rate 1/2. Failing this would indicate a training problem, not a scientific result. The main experiment then sweeps feedback signal-to-noise ratio over 30, 20, 10, and 5 dB and the price $\beta$ over 0, 0.5, and 1. At each point it plots block error rate against total rate for our policy, tuned DeepVLF with coded mask, the confirmation-phase scheme, and GBAF. It also reports the desynchronization rate, meaning how often the two ends disagree about which groups are still active.

The decisive check has two parts. The gap between our policy and tuned DeepVLF with coded mask should be near zero with perfect control and should grow as feedback degrades. Removing the belief network should erase most of that gap. The second experiment repeats the sweep on the hidden-state Gilbert–Elliott channel and on 5G NR-CDL fading at 1 and 10 m/s, where GBAF reports only "slight degradation" without numbers. It asks whether inferring the channel state yields further gains. A null result here would still be informative: it would show that memoryless-trained stopping policies transfer to fading channels, and that the benefit of the POMDP view comes entirely from noisy control signaling.