Shared Hidden Codes: Semi-Supervised Training of Binary Networks with Layer-Wise Targets

Networks whose hidden units output only one or two bits are cheap to run on edge hardware, but they are hard to train. Each hard threshold has zero gradient almost everywhere, so ordinary backpropagation has to be patched with the straight-through estimator: in the backward pass the step function is treated as if it were the identity. The best tools for learning from few labels are built on that same backpropagation. Methods such as FixMatch push every unlabeled image's prediction toward agreement across augmented views by sending gradients through the whole network. Low-bit networks are therefore being asked to learn from scarce labels with a biased gradient. We propose a different route.

The proposal treats training as auxiliary-variable decoupling, also called layer-wise target setting. Instead of backpropagating, we give every training image an explicit discrete target code at each hidden layer and train each layer separately to hit its target. The key move for unlabeled data is to keep a single stored code per unlabeled image, shared by all its augmented views. Agreement across augmentations then holds by construction at every depth rather than only at the output. Because the codes are bits, storing one per unlabeled image becomes affordable. That storage cost is what has kept auxiliary-coordinate methods out of the regime of many unlabeled images.

Core Claim

For hard-threshold networks (1-bit and 2-bit activations) trained from scratch with few labels, we claim that alternating between two steps beats straight-through-estimator versions of FixMatch at matched compute. The first step chooses a shared binary hidden code for each unlabeled image across its views. The second step fits each layer to those codes. We further claim that the gain grows as activations get coarser, and that it comes from the hidden-layer targets rather than from the output pseudo-labels alone.

This claim is narrow on purpose. For ordinary full-precision ResNets, end-to-end FixMatch already works extremely well (FixMatch reports 94.93% CIFAR-10 accuracy with 250 labels). The classic argument for decoupling, that SGD stalls in deep plain networks, comes from a 10-layer MLP on MNIST without skip connections (Global Convergence of Block Coordinate Descent in Deep Learning, roughly 11% for SGD versus about 80% for block coordinate descent). Residual connections largely remove that problem. We therefore do not stake the proposal on vanishing gradients. We stake it on non-differentiability, where decoupling has a documented advantage.

Where This Sits

Auxiliary-variable methods split a deep network into shallow subproblems by adding per-example variables for intermediate activations and alternating between updating weights and updating those variables. The Method of Auxiliary Coordinates does this with a quadratic penalty (Distributed optimization of deeply nested systems). ADMM does it with multipliers and closed-form activation steps (Training Neural Networks Without Gradients). The online minibatch variant (Beyond Backprop: Online Alternating Minimization) supplies our streaming machinery. All of these assume a label for every example, or a pure autoencoder objective.

Two papers supply the method and the motivation. Deep Learning as a Mixed Convex-Combinatorial Optimization Problem trains hard-threshold networks by setting discrete hidden targets (FTPROP). It reports 64.3% versus 60.6% top-1 for a 2-bit ResNet-18 on ImageNet against a saturated straight-through estimator, with all targets derived from labeled losses. Optimizing affinity-based binary hashing using auxiliary coordinates shows that binary auxiliary codes can be optimized alternately with the functions that produce them. It also motivates our bit-balance constraint, though that paper is fully supervised.

The closest work to ours is LapTAO. It introduces auxiliary proxy labels for unlabeled points, but only at the output, for decision trees, and with a fixed graph-smoothness prior. We place the auxiliary variables inside a deep network, make them discrete, and tie them across augmentations rather than across graph neighbours. DecoupledNet splits a semi-supervised segmentation system into a classifier and a segmenter, but trains each with ordinary backpropagation. Its decoupling is modular rather than layer-wise.

Two papers constrain the design. Realistic Evaluation of Deep Semi-Supervised Learning Algorithms shows that well-tuned supervised baselines and ImageNet transfer (12.09% CIFAR-10 error, better than VAT+EntMin's 13.13% at 4,000 labels) shrink apparent semi-supervised gains. It also shows that tuning on large validation sets inflates results. FixMatch reports sensitivity to its 0.95 confidence threshold and to many-class settings.

Method

For layer $l$ of a network with hard-threshold activations, each labeled image $x$ receives a code $t_l(x)$ in the FTPROP manner. Each unlabeled image $u$ receives one stored code $t_l(u) \in \{-1,1\}^{d_l}$, where $d_l$ is the number of units in layer $l$. All $K$ augmented views $g_1(u),\dots,g_K(u)$ share that one code, with $g_1$ the weak view. Given the codes, the weight step trains each layer on its own:

$$\min_{W_l} \sum_{x \in \mathcal{D}_L} h\big(a_l(x;W_l),\, t_l(x)\big) + \sum_{u \in \mathcal{D}_U} \sum_{k=1}^{K} h\big(a_l(g_k(u);W_l),\, t_l(u)\big).$$

Here $a_l(\cdot;W_l)$ is the layer's pre-threshold response, given the thresholded output of the layer below. The function $h$ is the soft hinge loss used by FTPROP. At the top layer, $t_L(u)$ is the weak view's pseudo-label when its confidence clears the threshold. Otherwise the unlabeled image contributes only hidden-layer terms.

The code step sets each bit of $t_l(u)$ by majority vote over the views. Each vote is the sign suggested by the layer above, so a view votes for the bit value that lowers the next layer's loss. A persistence penalty against flipping the previously stored bit damps the oscillation that drives confirmation bias, much as a slowly averaged teacher network does. A per-unit balance constraint keeps each bit near half on and half off across a minibatch. Without it, every unlabeled code could collapse to a constant pattern that satisfies agreement trivially.

The design keeps both kinds of SSL signal in a single network: agreement across views lives in the shared code, and pseudo-labeling lives at the output layer. It does this without any gradient passing through a threshold.

Memory follows from the bit width. A CIFAR-scale ResNet-18 has on the order of $10^6$ activation units per image. At one bit each that is roughly 125 KB per image, or about 6 GB for 50,000 unlabeled images. That fits in host memory and streams through the online minibatch scheme of Beyond Backprop. If needed, codes can be stored only at block boundaries. Storing the same quantities as floats would be 32 times larger, which illustrates the memory barrier those papers report.

Research Program

The work proceeds in three stages. First, we build the alternating trainer on 1-bit and 2-bit ResNet-18 for CIFAR-10. We start from the FTPROP target heuristic, add shared unlabeled codes, persistence and balance, and add confidence-gated top-layer pseudo-labels. Second, we analyse the dynamics. We measure how often bits flip per epoch, how often shared codes agree with each view's actual activation, and whether pseudo-label errors accumulate faster or slower than under the straight-through-estimator versions. Third, we run the stretch case: a 2-bit ResNet-18 on ImageNet with the 10% labeled split. There, FTPROP's supervised gap over the straight-through estimator was largest.

We will not attempt a convergence theorem. Hard thresholds and moving codes break the smoothness assumptions behind existing block-coordinate guarantees, so we treat convergence as an empirical question.

Baselines

The naive reference is a supervised-only low-bit ResNet-18 trained on the labeled subset with a straight-through estimator and strong augmentation. It is tuned with the same small validation set as every other method, following the protocol of Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. A second naive point is supervised FTPROP on labels only, which isolates what unlabeled data adds within the target-setting family.

The strongest direct competitor is FixMatch applied to the same quantized network, with a straight-through estimator through every threshold (FixMatch). It is the strongest because it is the best-known recipe for few labels and needs only a change of activation function. As a reference, FixMatch reports 94.93% accuracy with 250 CIFAR-10 labels and 28.54% top-1 error on 10%-labeled ImageNet, both at full precision. We found no reported result on Lacuna for FixMatch on 1-bit or 2-bit networks, so we will produce that number ourselves. It will use the same architecture, augmentations, unlabeled-to-labeled ratio, number of passes over the data, and hyperparameter search budget as our method. Beating it means higher mean test accuracy over five seeds at 40, 250 and 4,000 labels, with a margin larger than the seed spread, at matched wall-clock time.

The full-precision FixMatch number also serves as a ceiling. Closing the gap to it is the practical goal.

The third baseline is the most dangerous. We train a full-precision FixMatch teacher, then train the low-bit student with straight-through quantization-aware training on the teacher's outputs for all images. We give it the extra compute of training the teacher.

The ablation that isolates our key ingredient keeps everything identical but removes the shared hidden codes. Under this ablation, unlabeled images supply only top-layer pseudo-labels, and hidden targets for them come from each view independently. A second ablation ties codes across views but drops persistence. A third drops the balance constraint, to check for collapse.

Why not just…?

Why not just run FixMatch with a straight-through estimator on the binary network? This is the natural first try. The only evidence on our side is indirect: in the supervised setting, target setting beat a saturated straight-through estimator by 3.7 points top-1 on a 2-bit ResNet-18 (Deep Learning as a Mixed Convex-Combinatorial Optimization Problem). Biased gradients may matter even more when the training signal is itself a noisy pseudo-label. Whether they do is exactly what the main comparison measures, at both 1 and 2 bits. If the margin does not widen as bit width shrinks, our explanation is wrong.

Why not just train a full-precision semi-supervised teacher and distill it into the binary network? This is a real threat, and it may win when compute for a full-precision teacher is available. We handle it in two ways. First, we run it as a baseline with its extra compute included. Second, we run our trainer with the same teacher supplying the top-layer targets. That isolates the question of whether hidden-layer target setting fits a binary student better than straight-through quantization-aware training does, given identical supervision. If distillation with a straight-through student matches our teacher-fed variant, the proposal's contribution shrinks to the teacher-free setting, and we will report it that way.

Why not just put the auxiliary variables at the output, as LapTAO does? Output-level proxy labels are simpler and memory-light (LapTAO). The no-shared-hidden-codes ablation is exactly this design transplanted into a deep binary network. If it matches the full method, then tying hidden codes across views adds nothing, and the idea collapses to FTPROP plus pseudo-labels.

Why not just fine-tune an ImageNet-pretrained model? On CIFAR-10, transfer beat every from-scratch semi-supervised method (Realistic Evaluation of Deep Semi-Supervised Learning Algorithms). We report a pretrained-then-binarized ResNet-18 fine-tuned on the labeled set as an additional reference. We frame our method as relevant where no related pretrained low-bit model exists, and we do not claim it beats transfer on CIFAR.

Proposed Experiments

The decisive experiment is CIFAR-10 at 40, 250 and 4,000 labels, with 1-bit and 2-bit ResNet-18 and five seeds. Every method is tuned on a validation set of a few hundred labeled images. We report test accuracy, accuracy per GPU-hour, and memory. Three outcomes would support the claim: our method beats straight-through FixMatch at both bit widths; the margin is larger at 1 bit than at 2 bits; and the shared-code ablation loses most of the gain. If the shared-code ablation performs as well as the full method, or straight-through FixMatch matches it at 1 bit, the core claim is refuted.

Secondary checks run CIFAR-100 at 2,500 labels, where FixMatch is known to struggle without distribution alignment. Bit-flip and pseudo-label-error curves test whether persistence of the shared codes reduces confirmation bias. If CIFAR results hold, we will run 2-bit ResNet-18 on 10%-labeled ImageNet against straight-through FixMatch. The supervised FTPROP figure of 64.3% top-1 on full ImageNet anchors what a 2-bit ResNet-18 can reach.