Generating Spiking Networks at a Chosen Energy Budget from Tokenized Weights

Spiking neural networks (SNNs) are networks whose neurons talk in sparse binary pulses instead of dense real numbers. They save energy in a simple way: each pulse costs one addition, and a neuron that stays silent costs almost nothing. That makes the number of spikes the main dial on energy. Today, getting an SNN at a particular point on the accuracy–energy curve means training it from scratch with a spike penalty, once for every budget. That training is expensive because the network must be unrolled over every simulation timestep.

This note proposes to treat trained SNNs as data. We would train a population ("zoo") of small ternary-weight SNNs. Each network is cut into one token per layer, and each token carries both the layer's weights and that layer's neuron settings (firing threshold, leak constant). A sequence model learns from this zoo and then, for a new task, generates SNN weights conditioned on a target spike budget. The claim is that one such generator can produce a whole family of SNNs that cover the accuracy–energy curve on a held-out event-camera task. After one or two epochs of fine-tuning, those networks should match a separately trained per-budget sweep at a fraction of the training compute, and beat the cheap trick of re-thresholding a single trained model.

Core Claim and Positioning

Weight-space learning already shows that tokenizing networks layer by layer makes them learnable objects. SANE encodes networks as sequences of layer or channel tokens in a Transformer autoencoder and samples useful warm-start weights; it reports 39.4% Tiny-ImageNet accuracy after one fine-tuning epoch and a 25% gain over earlier zero-shot initialization on small CNNs. Falk et al. show that diverse zoos transfer much better than single-dataset zoos: 72.1% versus about 29.3% zero-shot on EuroSAT, using a masked per-token normalization to handle padded layers. Both works supply our method, but both operate only on continuous CNNs and never condition on energy.

On the spiking side, three works constrain the design:

- Temporal-adaptive Weight Quantization shows that 1.58-bit ternary SNN weights lose only 0.22% ImageNet accuracy. - The Hardware Impact of Quantization and Pruning finds that on spiking accelerator hardware, ternary weights beat pruning. Pruning costs 45% more energy at iso-accuracy, because storing which weights were removed adds memory traffic.

  • Bal & Sengupta give the per-operation energy model: about 0.9 pJ per accumulate versus 4.6 pJ per floating-point multiply-accumulate in 45 nm.

Together these say that generated weights should be ternary at the source, not continuous weights rounded afterwards. That fixes our token design: ternary weights are discrete symbols, so they are modeled as categorical tokens rather than standardized Gaussian vectors.

What is new is the combination: a population of SNNs as the training data, neuron dynamics carried inside the tokens, natively discrete weights, and generation conditioned on a spike budget. The research file finds no prior work tokenizing SNN weights. The closest spiking "tokenization" papers tokenize input data, not weights; the training-free spiking Transformer conversion is one example.

I deliberately did not pursue predicting spike rate or energy from weights alone. When the data are available, running one calibration batch through the SNN measures spike rate directly and cheaply. A learned predictor would answer a question nobody needs answered.

Research Program

The zoo. We first build the population, which does not exist yet. It holds roughly a few thousand small convolutional SNNs with leaky integrate-and-fire neurons and ternary weights. They are trained directly with spatio-temporal backpropagation plus threshold-dependent batch normalization (Zheng et al.). Training uses class subsets of DVS-Gesture and CIFAR10-DVS, plus frame-encoded CIFAR-10. The zoo varies:

  • seed,
  • firing threshold,
  • leak constant,
  • timesteps (T from 2 to 8),
  • the weight of a spike-rate penalty, so that members span a wide range of measured synaptic operations.

Batch normalization is folded into thresholds after training, so every member is inference-ready for hardware without normalization layers. This sidesteps the batch-norm recalibration step SANE needs, which neuromorphic chips cannot run.

Tokenization. Each layer is padded and chunked in the manner of Andreis et al.. Each chunk is a sequence of ternary symbols plus a small continuous field holding the layer's threshold and leak. A two-dimensional position code marks layer depth and chunk index. Permutation symmetry is handled with neuron-permutation augmentation, following Schürholt et al.. The loss is cross-entropy on the weight symbols and a masked regression loss on the neuron fields.

The generator. An autoregressive Transformer over layer tokens, in the style of Soro et al., is conditioned on two inputs:

  • a task embedding, computed from a small labeled support set of the new task;
  • a scalar budget $b$, the target number of synaptic operations per sample normalized by the network's maximum.

Estimated energy is $E \approx 0.9\,\text{pJ} \times N_{\text{SOP}}$, where $N_{\text{SOP}}$ is the count of spike-triggered additions (spikes times fan-out, excluding zero weights). The 0.9 pJ figure is the accumulate cost reported by Bal & Sengupta; we use it only as a relative proxy, not as a hardware measurement.

Baselines

The naive reference point is a randomly initialized ternary SNN of the same architecture. It is trained on the held-out task with the same one- or two-epoch fine-tuning budget at each target $b$. It shows how much any warm start matters at all.

The strongest existing method is the sweep: direct training with STBP and threshold-dependent batch normalization, run once per budget with a tuned spike-rate penalty and trained to convergence. This is the method people actually use, and it sets the ceiling. For reference, Zheng et al. report 96.87% DVS-Gesture accuracy for STBP-tdBN, and Amir et al. report 96.49% (10 classes) on TrueNorth. Neither number comes with a spike-budget curve, and the research file has no reported result for CIFAR10-DVS accuracy, so we re-run the sweep ourselves on identical architectures and data. Fairness rules:

  • Each sweep point gets the same hyperparameter search budget as the generator's fine-tuning stage.
  • We report GPU-hours both with and without the zoo's construction cost amortized across held-out tasks.

Beating the sweep means generated-then-fine-tuned models lie within 1 accuracy point of its accuracy–energy curve at every budget, at under a quarter of its total training compute on the new task.

The cheap competitor is post-hoc budget control. We fully train one SNN on the held-out task, then reach other budgets by scaling all thresholds up, or by per-layer threshold search on a calibration batch, with brief fine-tuning. To beat it, the generated family must dominate its accuracy–energy curve at low budgets, where heavy re-thresholding should starve deep layers of spikes.

The conversion competitor follows Liu et al., which reports 94.75% CIFAR-10 at T=2 with ResNet-18. We train an ANN on the held-out task, convert it with calibrated thresholds and unipolar neurons, ternarize, and fine-tune. We restrict to unipolar neurons because their negative-spike neurons do not run on standard neuromorphic chips.

Three ablations isolate the key ingredients:

  • an unconditional generator (SANE-style warm start, budget enforced only during fine-tuning);
  • continuous tokens with Falk-style normalization, ternarized after generation;
  • tokens without the threshold and leak fields.

Why not just…?

Why not just train one SNN and raise its thresholds to hit lower budgets? This is the most serious threat, because it costs one training run. Our bet is that budget-appropriate networks differ in where spikes are spent across layers, not just in how many, and a global threshold shift cannot redistribute them. The post-hoc baseline, including per-layer threshold search, tests this directly. If it matches the generator within noise, the conditioning idea is wrong and we report that.

Why not just convert a trained ANN? Conversion needs a fully trained ANN on the new task, often needs several timesteps to stay accurate, and its best low-latency variants depend on retrained clipping and negative spikes (Liu et al.). It may still be a strong warm start, so it is a baseline rather than a dismissed alternative. If converted-plus-fine-tuned models match the generator at equal compute, the generator's only remaining advantage is producing many budgets at once.

Why not just interpolate or average zoo members trained at different budgets? Falk et al. show that averaging aligned networks degrades sharply as more models are combined. Threshold nonlinearities and ternary rounding should make this worse for SNNs. We include aligned two-model interpolation at matched budgets as a small check.

Why not just tokenize with continuous weights like existing weight-space models? The ternarize-after ablation answers this. The hardware evidence that ternary weights beat pruning for energy suggests that rounding after the fact costs both accuracy and sparsity, but that has to be measured.

Decisive Checks

Held-out tasks are unseen class splits of CIFAR10-DVS and DVS-Gesture recordings under a lighting condition withheld from the zoo. Spiking Heidelberg Digits serves as a far-transfer stretch with a matched 1D architecture.

For each method, the deliverable is accuracy against measured synaptic operations at five target budgets, together with training GPU-hours. We also report:

  • Calibration of the generator: the correlation between requested $b$ and measured operations on held-out tasks. A generator that cannot hit its own budget fails regardless of accuracy.
  • Weight sparsity: the fraction of zero weights, which drives memory fetches on hardware.

The result that would confirm the proposal is the generated family dominating post-hoc re-thresholding at the two lowest budgets while matching the full sweep at a quarter of its compute. The result that would kill it is re-thresholding or conversion matching the generator at equal cost.