On a broadcast channel, one transmitter sends two messages at once to a strong receiver and a weak receiver. Codes built from linear algebra over bits (linear codes, where any XOR of valid codewords is again a codeword) can do no better there than time sharing, meaning simply alternating between the two users. The replica analysis of Statistical mechanics of broadcast channels using low-density parity-check codes proved this for linearly combined LDPC codes on binary symmetric channels. Doing better requires "cloud" or superposition codebooks, which are non-linear over bits. Neural encoders, meanwhile, find non-linear codes that beat linear ones on point-to-point channels (KO codes, DeepPolar). But their training is a black box, and it is fragile enough that DeepPolar needs a hand-tuned curriculum to converge. Nobody knows whether stochastic gradient descent (SGD) reliably finds codes beyond the linear ceiling, how long that takes, or what it depends on.
We propose a solvable model for this question. The model is a high-dimensional encoder made of sign units, sent over a binary broadcast channel. We will reduce its training to a handful of summary statistics ("order parameters"), in the teacher-student tradition of Saad and Solla. These statistics feed into a replica calculation of what rate the code reached at each moment of training. The result is a predicted rate trajectory: a curve showing when, and whether, training crosses the time-sharing line.
Core Claim
Consider an encoder whose output bits are signs of weighted sums of both messages' bits, $x_j = \mathrm{sign}(a_j\cdot m_1 + b_j\cdot m_2)$. These are threshold functions, not XORs. They are therefore outside the linear-code class that Nakamura et al. showed is capped at time sharing, and they are exactly the kind of function gradient methods find easy.
Our central quantity is the mixing angle of each encoder unit, $\theta_j = \arctan(\lVert b_j\rVert/\lVert a_j\rVert)$. It measures how much of the unit is devoted to each message. Units with $\theta_j\in\{0,\pi/2\}$ carry one message only, which reproduces time sharing. Units at intermediate angles form cloud structure.
We predict three things. First, training from small random weights passes through a symmetric plateau in which all units share a common mixing angle; this is the analogue of the plateau at generalization error 0.0203 that Saad and Solla found for three hidden units. Second, symmetry breaks into a two-population state: "cloud-center" units weighted toward the weak user's message and "satellite" units that mix in the strong user's message. Third, whether this break happens before training settles on the time-sharing configuration depends on the gap between the two channels' crossover probabilities and on the learning rate. The dependence should produce a sharp boundary in the (channel gap, learning rate) plane, which the theory predicts and experiments can test.
The novelty is linking two things that have not been linked. Dynamic order parameters from learning theory are plugged into the equilibrium free energy of coding theory. This turns "can SGD escape the linear ceiling?" into a question with a computable answer.
Research Program
Model. Messages $m_1\in\{\pm1\}^{k_1}$ and $m_2\in\{\pm1\}^{k_2}$ are encoded into $n$ bits by $n$ sign units. The limit is $k_1, k_2, n\to\infty$ at fixed rates. Each receiver sees the codeword through its own binary symmetric channel, and the weak receiver's channel is a degraded version of the strong one's. Each receiver has a trainable decoder: a single hidden layer mapping received bits to soft message estimates. The loss is the squared reconstruction error of each receiver's own message.
Projections of long $\pm1$ message vectors are asymptotically Gaussian. This is what lets the Gaussian-expectation machinery of Arnaboldi et al. apply even though the data are discrete.
Handling the non-differentiable sign. We train the sign units with the straight-through estimator, which treats the sign function as the identity when passing gradients backward. This lets us reuse the isotropy closure and ODE derivation of Ichikawa, Kashiwamura and Sakata. That work reports its ODEs match simulations with relative errors often below 1%, and that quantization raises the maximum stable learning rate by 35%. We will test whether that stabilizing effect carries over to encoders.
Order parameters. We track four families: the encoder's Gram blocks with respect to each message (which give the mixing angles), encoder-encoder overlaps, decoder-encoder alignments, and the population fractions of units at each angle.
Linking training to achieved rate. At any training time $t$, the order parameters define a code ensemble. We evaluate that ensemble's achievable rate pair with a replica-symmetric free energy written in the spin-glass language of Nakamura et al. This gives a trajectory $(R_1(t), R_2(t))$ that can be compared directly against the time-sharing line and against the three decoding phases that paper identifies: both receivers succeed, one succeeds, or neither does. We conjecture that the one-receiver-succeeds phase appears as a transient during training.
Deliverables. The program will produce the reduced ODEs, the plateau-escape time as a function of learning rate and channel gap (from linearizing the ODEs around the symmetric fixed point), and the predicted boundary between escape and entrapment.
Risk and fallback. The main risk is that decoder co-adaptation breaks the ODE closure. In that case we freeze the decoders to their Bayes-optimal posterior mean under the current order parameters, which is a standard reduction. That still yields the encoder dynamics, which carry the central claim.
Baselines
The naive reference is time sharing: two good point-to-point codes alternated in time. By Nakamura et al., this is also the ceiling for any linear code on this channel. A learned code "beats linear" only if it achieves a rate pair strictly outside this line at matched error rate, measured in simulation at finite $n$ and not just in theory.
The strongest structured linear competitor is the linearly combined LDPC scheme of Nakamura et al. with their parameters $C_1=C_3=3$, $C_2=4$, decoded by belief propagation (the standard message-passing decoder for sparse codes). That paper reports this scheme beats time sharing at finite length under belief propagation, while its optimal-decoding limit stays at time sharing. No reported numerical bit-error-rate result was found on Lacuna for this benchmark, so we will re-run it at the same $n$, channel pairs, and decoding iterations.
The strongest learned competitor is an end-to-end neural autoencoder in the KO/DeepPolar style, trained without our theory and given the same parameter count, training samples, and hyperparameter search budget as our model. The only broadcast-relevant number in the file is KO codes' roughly 0.7 dB gain over Polar codes at a bit error rate of $10^{-4}$, which is point-to-point. No reported result was found on Lacuna for learned codes on the broadcast channel.
A linearity ablation trains the encoder constrained to binary parity-check or generator matrices, using the gradient-quantization optimizer of Learning Linear Block Codes With Gradient Quantization. That paper reports learned codes matching CCSDS LDPC at 5 decoding iterations, 40 times fewer than the standard, for (128,64) codes. The theory predicts this ablation cannot exceed time sharing, which isolates non-linearity as the active ingredient.
A mixing-angle ablation pins every unit to $\theta_j\in\{0,\pi/2\}$. This isolates cloud formation, as opposed to mere sign quantization, as the cause of any gain.
Why not just…?
Why not just train an autoencoder and sweep hyperparameters? This is a real threat to the "does it escape" half of the question, since sweeps alone could show that escape happens. They cannot say where it fails or why. To make the theory earn its place, we derive the escape boundary and plateau lengths first. We then test them on held-out channel pairs and learning rates that were never used for fitting. If the predictions are no more accurate than interpolating the sweep, the theory adds nothing and we will report that.
Why not just use a hand-designed superposition code? Classical cloud-center constructions do beat time sharing, and we include one as a reference. Our target is different: we want to understand when gradient training discovers such structure. That understanding is what KO codes and DeepPolar currently lack when they tune curricula by trial and error.
Why not analyze training in the lazy/NTK regime? In the neural tangent kernel (lazy training) regime, the network's features barely move from initialization. Classifying High-Dimensional Gaussian Mixtures shows that kernel methods of this kind linearize in high dimensions and fail on XOR-structured data that feature-learning networks solve. Cloud formation is a symmetry-breaking feature-learning event, which lazy analysis by construction cannot capture.
Why not optimize linear codes better, e.g. with gradient quantization? Nakamura et al.'s result bounds all such codes at time sharing under optimal decoding. Better linear optimization improves finite-length belief-propagation performance, not the ceiling. The linearity ablation above measures exactly this.
Proposed Experiments
Calibrate the ODE solver. We first reproduce Saad and Solla's plateau value of 0.0203 for three hidden units, and the sub-1% ODE-versus-simulation agreement reported by Ichikawa et al.
Test the theory in the proportional limit. For binary symmetric channel pairs with crossover probabilities spanning small to large gaps, we simulate straight-through SGD at $k$ from 250 to 2000. We compare the empirical mixing-angle histograms, plateau exit times, and escape/entrapment outcomes against the ODE predictions.
Test finite blocklengths. At $n=64$, 128, and 256, we measure each receiver's bit and block error rates against time sharing, the Nakamura LDPC combination, the unconstrained autoencoder, and both ablations. We also check whether initializations or learning-rate schedules chosen from the predicted boundary cut the number of training steps needed to beat time sharing, relative to the tuned autoencoder.
Decisive check. The proposal succeeds if three things hold. The predicted escape boundary must classify held-out training runs correctly. The unconstrained encoder must exceed time sharing at finite $n$ while the linear ablation does not. And the mixing-angle ablation must remove the gain.
References
- Statistical mechanics of broadcast channels using low-density parity-check codes
- KO codes: inventing nonlinear encoding and decoding for reliable wireless communication via deep-learning
- DeepPolar: Inventing Nonlinear Large-Kernel Polar Codes via Deep Learning
- Dynamics of On-Line Gradient Descent Learning for Multilayer Neural Networks
- From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks
- High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator
- Learning Linear Block Codes With Gradient Quantization
- Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed