Where Do Balanced Bits Go Under Clutter? Recurrent Backbones for Maximum-Entropy Hashing

Image hashing compresses each image into a short binary code, such as 32 bits, so that a database can be searched with cheap bit operations instead of floating-point distances. The best hashing methods force every bit to split the data 50/50, a rule called bit balance, because a bit that is almost always 1 carries no information. Every such method sits on a purely feedforward image network, one that processes the image in a single bottom-up pass. That pairing has a hidden cost when objects are occluded or cluttered. A single-pass network cannot cleanly separate target from debris, as recurrent-vision work shows. Yet the balance rule still demands that each bit split the data in half, so it will split on whatever varies most, and under clutter that may be the debris rather than the object.

We propose to test this directly. We will measure how much of each balanced bit's information is about the clutter rather than the object, and ask whether replacing the feedforward backbone with a recurrent one moves that information back onto object identity. A recurrent network is one with lateral and top-down loops that re-process the image over several steps. Because a recurrent hashing network produces a valid code at every step, the same design also gives a cheap, label-free stopping rule: halt when the bits stop flipping.

Core Claim

Under occlusion and clutter, a hard bit-balance constraint on a feedforward backbone wastes part of the bit budget encoding nuisance structure. A recurrent backbone with the same parameter count, trained with the same constraint, reallocates that entropy toward object identity over its time steps. The benefit should be concentrated in cluttered inputs and nearly vanish on clean ones. Two information quantities make the claim testable. Write $b_i$ for bit $i$, $y$ for the target object class, and $c$ for the known nuisance variables of a procedurally generated scene (debris identities and positions). The clutter share of a bit is

$$\rho_i = \frac{I(b_i; c \mid y)}{H(b_i)},$$

that is, the fraction of the bit's entropy explained by clutter once the object class is known. The prediction is that average $\rho_i$ is high for feedforward balanced hashes under heavy debris and falls with recurrent time steps. Retrieval mean average precision (mAP), the standard ranking-quality score, should rise correspondingly.

Relation to Prior Work

The hashing side supplies the method. Deep Unsupervised Image Hashing by Maximizing Bit Entropy replaces the sign function with a parameter-free layer that assigns +1 to the top half of a mini-batch and −1 to the bottom half. This enforces exact balance, and gradients pass through with the straight-through trick. We reuse this layer unchanged, so any change we observe is attributable to the backbone. Two softer ways of encouraging balance serve as comparison points rather than ingredients. Deep Hashing for compact binary codes learning adds loss penalties for balance and bit independence. Embarrassingly Simple Binary Representation Learning adds a KL pull toward a fair-coin prior inside an information bottleneck. All three evaluate only clean benchmarks on feedforward backbones and never ask what the balanced bits encode when inputs are ambiguous.

The recurrence side motivates the proposal. Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition defines bottom-up (B), lateral (BL), top-down (BT) and full (BLT) recurrent networks. It shows that BLT reaches about 9.3% error on Digit Debris with 50 fragments, against 29.6% for a parameter-matched wider-kernel feedforward control. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior finds that primate object solutions for hard images arrive about 30 ms later and are not captured by feedforward models. This evidence shows that single-pass features are degraded on exactly the inputs we target. None of these works binarizes anything.

Two papers are closest to our proposal. Deep Reinforcement Learning for Image Hashing uses recurrence, but only in a bit-by-bit RNN readout over a frozen feedforward VGG-19. It has no balance mechanism and no corrupted-input tests. Our recurrence lives inside the image network, where clutter has to be resolved. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision halts a recurrent network when the class-probability entropy falls below a threshold. That rule needs a classifier head. Our stopping rule uses bit stability, which exists even for unsupervised codes.

One cited result constrains us. On clean ImageNet, the same speed-accuracy paper reports only 64.37% versus 62.68% top-1 for recurrent versus depth-matched feedforward networks. We therefore predict, and will test, that recurrence buys little on clean hashing benchmarks. A large clean-data gain would actually undercut the clutter-specific mechanism we claim.

Research Program

We will build B, BL, BT and BLT backbones following Spoerer et al., unrolled for $T=4$ (and up to 8) steps. Each step's top representation passes through a shared linear projection and a Bi-half layer, giving a $k$-bit code at every step. Training applies the hashing loss at every step, with later steps weighted more heavily. We use a supervised pairwise similarity loss as the main setting and the original unsupervised Bi-half objective as a secondary one. Bi-half's batch dependence at test time (an open problem the file notes) is handled by freezing per-bit median thresholds as running statistics from training. We report the train/test balance gap this introduces.

The analysis estimates the following quantities, all on held-out data, at each time step and bit length: per-bit entropy $H(b_i)$, pairwise bit mutual information as a redundancy measure, target information $I(b_i;y)$, and clutter share $\rho_i$. These are computable exactly because Digit Debris and our synthetic CIFAR occlusions are generated with known nuisance variables. This analysis directly addresses the open question of whether top-down feedback makes bits more correlated by enforcing global coherence, or less correlated by disentangling target from clutter.

For the stopping rule, let $f_t$ be the fraction of bits whose sign changes between step $t-1$ and step $t$ for one image. Inference halts at the first $t$ with

$$f_t < \tau,$$

and we sweep the threshold $\tau$ to trace mAP against the average number of steps.

Baselines

The naive reference is a standard feedforward B network with sign binarization, a tanh relaxation, and no balance term. It shows how much code collapse occurs at all under clutter. The core feedforward reference is the same B network with the Bi-half layer, which isolates the backbone effect.

The strongest competitors are the parameter-matched feedforward controls from Spoerer et al. These are B-K, with larger kernels, and B-F, with more feature maps, plus a depth-matched B-D in the style of the speed-accuracy paper. B-D gives the feedforward model the same number of layer applications that the unrolled recurrent model performs, so compute is matched as well as parameters. All of them get the Bi-half layer, identical training data including the same debris and occlusion distributions, and the same hyperparameter search budget, which we fix as an equal number of trials per model. Beating them means higher mAP at matched bits and matched multiply-adds on the heavy-clutter split, together with a lower measured $\rho$. A mAP gain without a $\rho$ drop would not support our mechanism.

For the strongest published hashing method, the reference points are from different settings and backbones than ours. On clean CIFAR-10, Bi-half reports 56.10% mAP at 16 bits (unsupervised) in Deep Unsupervised Image Hashing by Maximizing Bit Entropy. JMLH reports 0.841 mAP at 32 bits (supervised) in Embarrassingly Simple Binary Representation Learning, and DRLIH reports 0.853 at 48 bits in Deep Reinforcement Learning for Image Hashing. We found no reported hashing result on Lacuna for Digit Debris or occluded CIFAR-10. We will therefore re-run JMLH's bottleneck and a DRLIH-style recurrent readout over a B backbone under our protocol rather than quote numbers across setups.

Two ablations target the key ingredient. One crosses balance mechanism (Bi-half, soft penalty, KL prior, none) with backbone (B versus BLT), to show whether the recurrent benefit depends on hard balance. Our account predicts that unbalanced codes suffer less from the clutter-share problem but collapse instead. The other compares BL against BT, to see whether lateral or top-down loops do the reallocation.

Why not just…?

Why not just train the feedforward hasher with occlusion augmentation? This is a real threat, but it is already built into our controls. Every feedforward model trains on the same cluttered distribution. In Spoerer et al., B-K was trained on debris and still sat at 29.6% error against 9.3% for BLT. If augmented feedforward Bi-half matches BLT on mAP and $\rho$ in our runs, the claim is falsified.

Why not just use a deeper feedforward network? Unrolled recurrence is weight-shared depth, so the depth-matched B-D control is the honest comparison. The clean-ImageNet gap reported for BL over B-D is small (64.37% vs 62.68%). We therefore expect parity on clean data and require the advantage to appear specifically as clutter increases. If B-D tracks BLT across clutter levels, recurrence adds nothing beyond depth, and we will say so.

Why not just average features over test-time crops? Multi-crop averaging spends extra compute without any hypothesis about where the object is. Look and Think Twice shows that class-guided feedback reduced ImageNet localization error from 44.6% to 38.8% over its baseline. We run a feedforward Bi-half model with $T$ crops, so that compute equals the BLT unroll, as an explicit baseline.

Why not just add bits? Longer codes could absorb the clutter-encoding bits. We report mAP across 16, 32 and 64 bits and the bit length at which feedforward matches recurrent. If feedforward needs roughly $2\times$ the bits, that is itself the storage cost of the feedforward constraint.

Proposed Experiments

The primary benchmark is Digit Debris, regenerated procedurally with 0 to 50 fragments and with SNR noise sweeps, using target-class retrieval at 16, 32 and 64 bits. The second is CIFAR-10 with synthetic occlusion by patches from other CIFAR images, with known patch sources. We also report clean CIFAR-10, Flickr25k and NUS-WIDE to confirm parity rather than gain on uncorrupted data. Metrics are mAP, Precision@k, per-bit entropy, pairwise bit mutual information, $I(b_i;y)$, $\rho$, flip rate, and multiply-adds per query.

Three outcomes decide the proposal. First, the gap between BLT and B-D in mAP and in $\rho$ must grow with the number of debris fragments and be near zero on clean data. Second, the clutter share $\rho$ must decrease monotonically across BLT time steps while per-bit entropy stays at about 1 bit, which would show reallocation rather than collapse. Third, the bit-flip stopping rule must reach the full-$T$ mAP with fewer average steps than the class-entropy rule from the speed-accuracy paper, and with no classifier head in the unsupervised setting. If the first check fails, the project reduces to a negative result: balance constraints are robust to clutter on feedforward backbones. That finding would still be worth publishing, since no prior work has measured it.