Event cameras do not record images. Each pixel independently reports a tiny tuple (position, microsecond timestamp, sign of brightness change) whenever it sees a change. Models for this data disagree on how to read it. Convolutional networks stack events into frames with a chosen number of time bins. Graph networks treat each event as a node. Spiking networks integrate events over simulated time. Sequence models embed each event like a word. Dataset distillation compresses a large training set into a handful of synthetic examples that still train a good model. It has been worked out for images, videos, and point clouds, but not for event streams. The obvious way to bring it here is to distill voxel grids, meaning frame stacks at a fixed bin count. That is exactly the wrong move: it bakes one temporal discretization into the synthetic data, so the result only suits models that read events the same way.
We propose to store the distilled set as native events: continuous $(x, y, t)$ coordinates plus a soft polarity. We optimize it so that it matches the real data under a randomly sampled discretization at every step. The bet is that one small synthetic event set, matched in expectation over many ways of turning events into tensors, will train frame-based CNNs, graph networks, spiking networks, and event-sequence Transformers alike. It should also make student accuracy insensitive to the bin count and window each architecture chooses. It keeps the low synthesis cost of distribution matching, because it never unrolls an inner training loop.
Core Claim
Cross-architecture failure on event data is mostly a mismatch in how time is discretized, not a mismatch in network weights. Distilling at the level of continuous events, with the discretization randomized during matching, should close most of the gap between distilled and full-data training across four architecture families. It should do so at a single synthesis cost of roughly an hour, and it should flatten the tuning surface over temporal bin count.
The claim is falsifiable in a specific way. If a random or teacher-selected subset of real event streams at the same event budget does as well, the synthesis step is not earning its keep.
What Prior Work Supplies and Leaves Open
The matching machinery comes from distribution matching.
- M3D matches real and synthetic features under a Gaussian-kernel maximum mean discrepancy (a distance between distributions that compares all their moments, not just means). It reports 73.4% on ImageNet-10 at 10 images per class in 1.1 hours, against 70 hours for bi-level IDC. It transfers from ConvNet-3 to ResNet-10 and DenseNet-121. It works on static images and relies on a sensitive kernel bandwidth.
- SADM handles unordered point clouds by sorting per-channel features before matching. Its sets, distilled on PointNet, transfer to DGCNN and Point Transformer. It also cuts ShapeNet distillation from over 7 hours to 0.35 hours.
We reuse the kernel matching of the first and the sorting trick of the second. The events $(x, y, t)$ form an unordered 3D point set, so sorting applies directly. Neither paper touches time.
The closest intersection work is ProtoBlend. It selects teacher-scored video clips, clusters them, and blends pixels. This gives 41.0% on MiniUCF at 10 videos per class in under an hour versus over 20 hours for VDSD, with transfer between CNN+GRU and CNN+LSTM. It operates on fixed 16-frame clips and depends on a VideoMAE teacher. We take its lesson that selection plus cheap mixing is a strong, cheap baseline, and we test it directly (below). We do not take its frame-based representation.
DDP stores coarse anchor point sets and learns convex blends, reaching 87.7% on ModelNet10 at one sample per class. It shows that parameterized point sets can transfer to PointMamba. Its reported collapse below 128 anchor points warns us that event budgets per stream have a floor we must measure.
On the event side, three papers motivate the problem and supply the rendering.
- EST gives a differentiable path from events to voxel grids through a learned temporal kernel. It is our differentiable renderer. Its own dependence on preset bin count and window is the tuning problem we target.
- AEGNN treats events as a spatiotemporal graph and reaches 94.5% on N-Cars.
- Event2Vec embeds inter-event intervals for a Transformer.
Together these show that the field has no agreed discretization. A distilled set tied to one would be a set for one architecture. STSC-SNN supplies a strong spiking student (81.4% on CIFAR10-DVS, 98.96% on DVS128 Gesture). It explicitly never asks whether spiking networks can learn from distilled data.
Method
For each class $c$, the synthetic set $S_c$ holds $m$ streams of $n$ events each. Every event has learnable continuous $(x, y, t)$ and a polarity logit. At each step we sample a discretization $\rho$: a bin count between 3 and 15, a window length, and an EST-style kernel shape. We also sample a randomly initialized encoder $f_\theta$. The encoder comes from one of two families. One is a small ConvNet over the rendered grid $R_\rho(\cdot)$. The other is a PointNet-style encoder over raw events with SADM channel sorting, for which $R_\rho$ only rescales time. We then minimize
$$\mathcal{L}(S) = \sum_c \mathbb{E}_{\rho,\theta}\, \mathrm{MMD}_k^2\big(f_\theta(R_\rho(S_c)),\, f_\theta(R_\rho(D_c))\big),$$
where $D_c$ is a minibatch of real class-$c$ streams and $k$ is a Gaussian kernel. The kernel bandwidth is set by the median pairwise distance at each step, so that no bandwidth must be tuned.
Gradients reach timestamps because trilinear or kernel splatting is differentiable in $t$. Gradients reach polarity through its soft weight. Synthetic streams are initialized from ProtoBlend-style selected real streams, so the method is by construction a refinement of selection, and the ablations below measure what refinement adds. After distillation, polarities are hardened and events are exported as ordinary event files. Any downstream pipeline, including asynchronous graph and spiking ones, consumes them unchanged.
Research Program
We build the distiller and export synthetic sets at 1 and 10 streams per class, with the event budget per stream matched to the real-data median. We then train, with fixed recipes, four student families that never participated in matching:
- an EST+ResNet-34 frame model,
- an AEGNN-style spline graph network,
- STSC-SNN,
- an Event2Vec Forgetting Transformer.
The Transformer and the graph network are the real test. The graph family sees no graph during matching, and the Transformer family sees no sequence model.
Hyperparameter sensitivity gets its own analysis. For each student we sweep bin count and learning rate on a small grid. We report the best cell, the mean, and the spread of accuracy across cells. The spread measures how much re-tuning a practitioner would need.
Baselines
The naive reference points are two random real-data subsets. The first draws random real streams at the same stream count and total event budget. The second is a temporally subsampled version, where events are dropped uniformly to fit the budget, mirroring the uniform sampling Event2Vec uses to fit context limits. Both are trained with each student's standard recipe. Beating them by a clear margin, at least several points averaged over students at 1 stream per class, is the minimum bar.
The strongest competitor is an event-domain port of ProtoBlend. A full-data EST+ResNet teacher scores candidate time windows, K-means allocates slots across teacher-feature clusters, and "blending" is the union of two selected streams' events with blended soft labels. Event superposition is the natural event analogue of pixel blending. It is strong because it is non-iterative, uses only real events, and is therefore representation-agnostic for free. It gets the same teacher and the same per-student training, and we report its construction time next to ours. On video it reports 41.0% on MiniUCF at 10 videos per class. For any event benchmark there is no reported result found on Lacuna for distillation of event-camera datasets, so this port is run by us. We also give our method's soft labels to the ProtoBlend port, so that label information is not a confound. Beating it means higher accuracy averaged across all four student families, and lower spread across the bin-count grid, at comparable wall-clock.
The third baseline is fixed-discretization distillation: M3D-style kernel matching on EST voxel grids at a single bin count (9), exported as grids. Grid-consuming students get it natively. Graph, spiking, and sequence students get events resampled from the grid. It is the "obvious" method.
The key ablation is our own method with $\rho$ frozen at one discretization. Everything else is identical, including event parameterization and export. It isolates whether randomizing time discretization, rather than simply storing events, drives transfer.
Full-data upper references come from the file: EST reports 92.5% on N-Cars and 81.7% on N-Caltech101, AEGNN 94.5% on N-Cars, and STSC-SNN 81.4% on CIFAR10-DVS and 98.96% on DVS128 Gesture.
Why not just…?
Why not just keep a subset of real event streams? Real streams are already continuous and architecture-agnostic, so this is the most serious threat. The answer from images is that distilled data beats coresets at tiny budgets. M3D's gains over plain distribution matching (69.9% vs 63.0% on CIFAR-10 at 50 per class) suggest that matched data carries more per sample. Whether that holds for sparse events is exactly unknown. The random-subset and ProtoBlend-port baselines settle it. Because our method initializes from selected streams, any failure to improve on them will be visible as a null effect of refinement, and we would report it as such.
Why not just distill voxel grids with M3D and convert? Grids fix the bin count, and EST documents that a mismatched bin count blurs fast motion or leaves voxels empty. Converting grids back to events recovers only bin-center timestamps, which should starve the graph and sequence students that read fine timing. The fixed-discretization baseline measures this loss directly.
Why not just use heavier augmentation, as in differentiable Siamese augmentation, instead of randomized discretization? Siamese augmentation randomizes spatial transforms, which fights encoder-specific shortcuts but not the temporal representation mismatch. We include spatial and temporal-jitter augmentation in the frozen-$\rho$ ablation. If that ablation matches the full method, randomized discretization is unnecessary and the claim fails.
Proposed Experiments
We run five benchmarks: N-Cars, N-Caltech101, CIFAR10-DVS, DVS128 Gesture, and N-MNIST. Budgets are 1 and 10 streams per class. For each budget we report accuracy for all four students under all five methods (the two random subsets, the ProtoBlend port, fixed-grid distillation, and ours), plus the frozen-$\rho$ ablation. Overhead is reported as wall-clock and peak GPU memory for synthesis, including teacher training where used. The reference points are M3D's 1.1 hours and ProtoBlend's under-1-hour construction.
Three checks decide the claim.
The first compares the ProtoBlend port and our method on the Event2Vec and AEGNN students. These are the architectures furthest from the matching encoders. If we do not beat the port there, the method adds cost without transfer.
The second is the spread of accuracy over the bin-count grid. It must be clearly smaller for our sets than for fixed-grid distillation. Otherwise the tuning claim fails.
The third looks at the time axis itself. We re-render our synthetic sets and the real subsets at progressively coarser time resolution and plot accuracy. If our sets' advantage persists at the finest resolution and shrinks at coarse resolution, the synthetic timestamps carry real temporal information. If the advantage is flat across resolutions, the gains are spatial and "event-level" distillation is a label on ordinary image distillation.
References
- M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy
- Dataset Distillation of 3D Point Clouds via Distribution Matching
- Efficient Video Dataset Distillation via Cluster-Guided Prototype Blending
- Parameterization-Based Dataset Distillation of 3D Point Clouds through Learnable Shape Morphing
- End-to-End Learning of Representations for Asynchronous Event-Based Data
- AEGNN: Asynchronous Event-based Graph Neural Networks
- Event2Vec: Processing neuromorphic events directly by representations in vector space
- STSC-SNN: Spatio-Temporal Synaptic Connection with temporal convolution and attention for spiking neural networks
- Dataset Condensation with Differentiable Siamese Augmentation