Is the Geometric Free Lunch Real? Auditing Simplex and Orthogonality Losses with Transfer Attacks

A small group of training losses claims something that adversarial training has never delivered: robustness to small malicious input perturbations, higher clean accuracy, and almost no extra compute. The strongest such claim comes from the Gaussian class-conditional simplex (GCCS) loss. It forces each class's internal representation toward one corner of a regular simplex, a shape whose corners are all equally far apart. The authors report about 85% CIFAR-10 accuracy under a 5-step projected gradient descent (PGD) attack, about 4% higher clean accuracy than cross-entropy, and only about 2% extra training time (Ali et al.). If that held, it would largely retire both the robustness-accuracy trade-off and the 2–10× cost of adversarial training. History argues for caution. Error-correcting output codes looked equally free, but a logit-targeted attack broke them with 98.6% success (Challenging ECOC). Evaluations limited to weak attacks routinely overstate robustness, "sometimes by more than 10%" (RobustBench).

This note proposes using adversarial transferability as the measuring instrument. Transferability is the tendency of an attack crafted on one model (the surrogate) to fool a different model (the target). Suppose a geometric loss truly widens decision margins. Then attacks transferred from an ordinary cross-entropy surrogate should do no better than direct white-box attacks. Suppose instead the loss merely makes gradients uninformative, which is called gradient masking. Then transferred attacks will beat white-box ones. There is a third possibility: the loss moves the target's decision boundaries away from those of standard models without adding true margin. In that case transfer attacks will fail while direct attacks succeed. Measuring the white-box/transfer gap, alongside an attack designed for simplex geometry, tells these three stories apart. It also tells practitioners which of the promised benefits (robustness, accuracy, low cost) is real and under which threat model.

Core Claim

The claim is that passive geometric losses (GCCS, standalone Orthogonal Projection Loss, Guided Complement Entropy) can be placed on a map with two axes: true margin and boundary decoupling from standard models. The map is built from the ratio between transfer-attack accuracy and strongest-white-box accuracy on the same target, at matched perturbation budget.

The file motivates a specific and falsifiable prediction. Charles et al. prove that for linear classifiers the max-margin solution gives a universal adversarial direction, one that fools every valid linear separator. They also show that attacks from more complex models transfer to simpler ones, but not the reverse. A loss that imposes fixed, equidistant class targets and linear Voronoi boundaries, meaning each class owns the region nearest its corner, pushes the final layers toward exactly this max-margin linear regime. We therefore predict that GCCS targets will be at least as susceptible to transfer from well-trained cross-entropy surrogates as cross-entropy targets are. If so, their reported PGD-5 robustness would come mostly from gradient masking rather than margin, and the clean-accuracy gain is the only benefit that survives. The proposal is designed so that the opposite finding, genuine margin or genuine decoupling, would be equally clear.

Relation to Prior Work

Three papers supply the defenses under audit. Each removes the inner attack loop of adversarial training, and none evaluates transfer.

GCCS fits latent Gaussians to simplex corners using a KL-divergence term and a kurtosis penalty, and reports only PGD-5.

Guided Complement Entropy flattens probabilities on wrong classes. It reports about 15% accuracy under the Momentum Iterative Method (MIM) on CIFAR-10, versus 0% for cross-entropy, and a lower Tiny ImageNet error (38.56% vs 39.54%).

Orthogonal Projection Loss pulls same-class features together and pushes different classes toward orthogonality. It was tested under AutoAttack only on top of full adversarial training, never standalone.

We reuse these losses as they are; our contribution is the measurement and one new attack.

Two papers motivate the transfer lens.

  • Charles et al. give the geometric theory. Their evidence comes from MNIST, linear models, and two-layer networks. We test it on deep networks whose last layers have been deliberately linearized by a geometric loss.
  • Song et al. show that one model can gain white-box robustness while becoming more exposed to transfer attacks. They find this when fine-tuning from a public backbone. This is exactly the kind of divergence our diagnostic reads. Their study used only single-step FGSM, and we use iterative and ensemble attacks.

Two papers constrain the design.

  • The ECOC break (Challenging ECOC) shows that output-geometry defenses must face attacks written against their own output structure. The research file notes that no analogous attack on simplex latent coordinates exists. We build one.
  • RobustBench supplies AutoAttack and establishes that adversarial training dominates standardized evaluation, at high compute and with poor calibration. It is our reference for what "real" robustness costs.

Research Program

Training the targets. We train target models under five losses: cross-entropy, GCCS, GCE, standalone OPL, and PGD adversarial training. Every model uses identical architectures, data augmentation, and epochs. Wall-clock training time is logged for each.

Attacking each target in three ways.

  • White-box. We run PGD with 5 and 50 steps, MIM, and full AutoAttack. AutoAttack combines two automatic-step-size PGD variants, the Fast Adaptive Boundary attack, and the query-only Square attack.
  • Transfer. Attacks are crafted on independently trained surrogates and transferred without any queries. Surrogates include cross-entropy models of the same and different architectures, an ensemble of such models attacked with momentum, and an adversarially trained surrogate. We also include a loss-matched surrogate trained with the target's own public loss but a different seed and architecture. This models an attacker who simply reads the paper.
  • Adaptive, latent-coordinate. This is our new attack, written specifically against GCCS. It ignores the softmax and directly minimizes the distance between the target's latent code and the nearest wrong simplex corner, with a margin term. It is the continuous counterpart of the bit-targeted ECOC attack.

Direct geometry measurements. Alongside attack success, we measure geometry directly. We record the cosine similarity between surrogate and target input gradients, the approach Charles et al. use to argue about shared adversarial directions. We also estimate the minimum boundary distance with a decision-only search that needs no gradients, so gradient masking cannot inflate it.

Placing each loss on the map. From these measurements we read off each loss's position on the two-axis map.

  • True margin: high white-box robustness, transfer accuracy no higher than white-box accuracy, and a large gradient-free boundary distance.
  • Masking: transfer or Square attacks beating gradient attacks, and robustness that does not fall to zero as the perturbation budget grows.
  • Decoupling: low gradient alignment with cross-entropy surrogates and transfer failure. This must hold even against the loss-matched surrogate for it to count as anything more than obscurity.

Beyond CIFAR. If any loss lands in the margin or decoupling region on CIFAR, we repeat the audit in the realistic threat model of Song et al.. There, a downstream model is briefly fine-tuned from a public backbone with the geometric loss, and the attacker holds that backbone.

Baselines

The naive reference point is a plain cross-entropy model. It fixes the floor for clean accuracy, white-box accuracy, and transfer accuracy, and it gives the natural transfer rate between two independently trained standard networks. That natural rate matters because any change of seed, architecture, or loss lowers transfer somewhat. A geometric loss must lower transfer by more than a different-seed or different-architecture cross-entropy target does before any decoupling claim is allowed.

The strongest existing method is standard PGD adversarial training plus OPL, as reported by Orthogonal Projection Loss. That paper reports CIFAR-10 AutoAttack robust accuracy of 44.04% for the standard defense, rising to 49.15% with OPL added. These figures come from the paper's own architecture and budget. We will reproduce both models on our backbone with the same epochs and augmentation, and report their GPU-hours next to everyone else's.

A passive loss "beats" this baseline in one of two ways:

  • It matches the baseline's AutoAttack accuracy within a few points and its transfer accuracy, at a fraction of the compute.
  • It achieves strictly higher clean accuracy with transfer robustness equal to adversarial training, which would be a meaningful narrower win.

The passive losses themselves are baselines against their own published numbers:

  • GCCS against its reported ~85% PGD-5 accuracy, ~4% clean gain, and ~2% overhead (Ali et al.);
  • GCE against its ~15% MIM accuracy (Chen et al.).

No reported AutoAttack result was found on Lacuna for standalone GCCS, GCE, or OPL. Producing those numbers is part of the contribution.

Two ablations isolate the key ingredient, which is the fixed equidistant geometry rather than incidental changes to the model's outputs.

  • Learned targets. GCCS with learned rather than fixed class centers. This tests whether the simplex itself matters.
  • Masking control. Cross-entropy with extreme label smoothing and logit rescaling, tuned to reproduce GCCS's output saturation without any geometric target. If this control reproduces GCCS's PGD-5 gains and its white-box/transfer gap, the gain is a masking artifact of output scale, not geometry.

All methods receive the same hyperparameter search budget, measured in trials per method, on a held-out validation split.

Why not just…?

Why not just run AutoAttack and skip the transfer machinery? We do run AutoAttack, and it will likely deflate the PGD-5 numbers. AutoAttack reports one worst-case number, though. It cannot say whether a defense that fails white-box still protects a private model against attackers who only hold public surrogates. That protection is the scenario where a cheap loss could still earn its place. AutoAttack also cannot separate the margin, masking, and decoupling explanations. The white-box/transfer gap and the loss-matched surrogate can.

Why not just use adversarial training, perhaps with fewer steps? Adversarial training is the honest reference, and RobustBench shows it dominates standardized robustness (RobustBench). It is also a real threat to the whole premise of passive losses. If every passive loss falls to the masking region, the conclusion is simply that nothing replaces the inner loop. That is a publishable negative result, and we will state it plainly. The comparison runs on matched hardware and reports cost, so the trade-off is explicit rather than assumed.

Why not just add a few secret tokens to a frozen model, as in Robustness Tokens? Robustness Tokens recovers DINOv2-Giant accuracy under white-box attack from 0.3% to 63.1% in 200 steps, keeping 85.2% vs 85.6% clean accuracy. By its own account, however, the defense depends on the tokens staying secret. Our loss-matched surrogate is exactly the test of whether a defense survives an informed attacker. Robustness Tokens also answers a different question: patching a frozen model, not whether a training loss changes the model's geometry. We treat it as out of scope for the from-scratch audit, but it enters the fine-tuning extension as a comparison point.

Why not just use error-correcting output codes, which also need no inner loop? They were broken by an attack aimed at their own output structure, with 98.6% success on CIFAR-10 and 0.993 wrong-class confidence (Challenging ECOC). That result is precisely why we build the latent-coordinate attack against GCCS rather than trusting standard attacks.

Proposed Experiments

Datasets. We use CIFAR-10 and CIFAR-100, where AutoAttack and RobustBench protocols apply, and Tiny ImageNet, where GCE reports clean-error gains.

Models and threat model. Architectures are ResNet-18 and a WideResNet for targets, with VGG-16 and a second ResNet seed among the surrogates. Attacks use ℓ∞ perturbations at budget 8/255, with a sweep up to budgets where accuracy must reach zero.

Metrics. We report clean accuracy, AutoAttack accuracy, best-transfer accuracy, the transfer-to-white-box ratio, accuracy under the latent-coordinate attack, gradient-free boundary distance, gradient cosine alignment, expected calibration error, and training GPU-hours.

Decisive checks. Four results would each change the conclusion.

  • Masking, confirmed. On any passive loss, transfer or Square accuracy falls below PGD accuracy, or the latent-coordinate attack drives accuracy near zero. Its white-box claim is then attributed to masking. If the label-smoothing control reproduces the effect, the attribution is confirmed.
  • Real margin. A loss keeps AutoAttack accuracy within a few points of adversarial training, shows no white-box/transfer inversion, and increases the gradient-free boundary distance. That would be strong evidence that the geometric free lunch is real.
  • Decoupling. Transfer failure holds against cross-entropy surrogates but disappears against the loss-matched surrogate. The loss then provides only obscurity.
  • The Charles et al. prediction. We will state whether the complexity-asymmetry prediction holds on deep networks. Cross-entropy-to-GCCS transfer at least matching cross-entropy-to-cross-entropy transfer would support it; the reverse would falsify it.