Private Temporal Marginals for Exemplar-Free Video Continual Learning

A model that learns new video categories one batch at a time forgets the old ones unless it can rehearse them. The usual fix is to keep a few old examples, or their internal feature vectors, and replay them. This fails wherever the videos come from people, such as patients, home cameras, or user devices, because each stored item is still a record of one specific person's clip. We propose a different kind of memory. Each individual clip is held only briefly, as a short-lived episodic trace. At the end of each learning stage, all clips of a class are collapsed into a small table of semantic statistics: the average and spread of their features in each time segment of the action, plus how often one segment's state follows another. Then every individual trace is deleted. Old classes are rehearsed by sampling feature trajectories from these tables.

The point is that this one operation does two jobs. Summing episodes over time and over instances is how the memory literature formalizes the step from episodic to semantic memory. A sum over many clips is also exactly the kind of quantity whose dependence on any single clip can be bounded and hidden with calibrated noise. That makes it compatible with differential privacy, a formal guarantee that the stored memory would look nearly the same whether or not any one video had been used. So the claim is that keeping the temporal shape of the average, rather than either individual clips or a single time-free class average, is what lets exemplar-free rehearsal work on motion-heavy video under a formal privacy guarantee.

Core Claim

We claim three things. First, time-sliced class statistics, privatized at the level of a single video, retain enough temporal structure to rehearse actions whose identity depends on motion direction or order. Second, the time-free class average cannot do this. Third, protecting stored per-video features to the same privacy level requires so much noise that they become useless. The test case is the comparison between Something-Something-V2 (SSV2), where order matters, and UCF-101, where appearance mostly suffices. If the temporal statistics do not beat a time-free average on SSV2, the idea is wrong.

Grounding in Prior Work

The formal starting point is the view from Tresp et al. and Daxberger et al.. They treat semantic facts as the result of summing time-stamped episodic events over time. That work operates on symbolic triples. We apply the same summation to continuous video features, and we sum over instances as well as time, because the instance sum is what produces the privacy property. TSM supplies a practical device: it slices a timeline and clusters entities with Gaussian mixtures into lasting memories. TSM does this for text dialogue, with an external database and no privacy accounting. We reuse the idea of slicing and soft clustering, and observe that mixture statistics are themselves sums of soft counts, so they can be privatized.

The closest prior work is ESSENTIAL. For video class-incremental learning, it pairs a sparse episodic buffer of 1–2 real frame features per video with learned semantic prompts that reconstruct dense temporal features through cross-attention. It reports 95.8% average incremental accuracy on UCF-101 using 9.7 MiB of memory, against 93.4% and 26.4 GiB for the dense-replay baseline PIVOT. ESSENTIAL motivates our work: it shows that sparse episodic plus abstract semantic memory is enough for strong rehearsal. It also constrains our design: it keeps per-video features permanently, and our method must give that up. We differ in that no per-video quantity survives past the stage in which it was seen.

Two privacy-motivated vision-language methods avoid storing pixels. CoMem stores concept graphs anchored by token-level features and rehearses with a graph-conditioned feature generator. SGP replays symbolic scene graphs from a small language model. Both describe static snapshots with no time dimension. Both also treat "no raw images" as the privacy criterion, without measuring leakage. CoMem's observation that unconstrained synthetic replay drifts off the data's natural range of features constrains our generator design. Tasai et al. argues that intermediate CT features are non-invertible and therefore compliant with medical privacy rules. We treat that as a hypothesis to test with attacks, not an assumption.

MemoryVLA and HyperMem show two other routes to combining episodic and semantic memory. MemoryVLA is a robot working memory that is wiped between episodes. HyperMem writes memory into the weights of a text model's feedforward layers. Neither consolidates across episodes under a privacy budget, and our design keeps MemoryVLA's "clear the episode" behavior as a feature rather than a limitation.

Research Program

The learner is a frozen pretrained video-language encoder with a lightweight trainable adapter and classifier. Rehearsal happens in the encoder's fixed output space, which means stored statistics do not go stale as training proceeds. This is a deliberate trade of some flexibility for stability, and we test a version with a drift-correcting projection.

Each clip $v$ of class $c$ is split into $K$ equal time slices. Slice $k$ is encoded and projected to a low-dimensional feature $\phi_k(v)$. The visual and the language embeddings of the clip's label or caption are concatenated, so replayed trajectories stay aligned with the text side of the model. Each feature vector is clipped to norm at most $B$. During a stage, these features live in an episodic buffer and are used normally. At the end of the stage, we compute and release the following for every class $c$ and slice $k$:

$$ \tilde S_{c,k} = \sum_{v \in c} \phi_k(v) + \mathcal{N}(0,\sigma^2 B^2 I), \qquad \tilde Q_{c,k} = \sum_{v \in c} \phi_k(v)\phi_k(v)^\top + \mathcal{N}(0,\sigma^2 B^4 I). $$

Here $\tilde S_{c,k}$ is the noisy sum of slice features, and $\tilde Q_{c,k}$ is the noisy sum of their outer products, from which a covariance can be recovered. We also release noisy counts of transitions between per-slice mixture components, using a small mixture model in the style of TSM. Removing one video changes each sum by a bounded amount. A noise scale $\sigma$ calibrated to that bound therefore gives a formal per-video privacy guarantee $(\varepsilon,\delta)$, accounted across all released quantities and all stages. After release, the buffer is deleted.

Rehearsal samples a trajectory for an old class in two steps. It first draws a path of mixture components from the transition counts, then draws features from each slice's Gaussian. CoMem's lesson about off-manifold drift motivates a confidence filter: the current classifier must agree with the sampled label before the trajectory is used.

Two weak points are clear in advance. Small classes such as those in HMDB51 (roughly a hundred clips) suffer the most from added noise, and we report results stratified by class size. Second-moment statistics in high dimension are noisy, so the projection dimension and the number of slices $K$ are the central hyperparameters. We sweep both, and choose them on held-out early stages only.

Baselines

The naive reference point is fine-tuning with no rehearsal, which shows how much forgetting there is to recover. The upper reference point is joint training on all classes, which shows how much is recoverable at all.

The strongest existing method is ESSENTIAL. On UCF-101 it reports 95.8% average incremental accuracy at 9.7 MiB, the best number in our file. No reported result was found on Lacuna for ESSENTIAL on SSV2, HMDB51, or Kinetics-400, so we will rerun it there. For fairness, all methods share the same frozen encoder and adapter, the same class splits and stage order, matched memory in bytes, and equal hyperparameter search budgets. We run ESSENTIAL in two forms: as published, which has no formal privacy, and with per-feature clipping and Gaussian noise tuned to the same $(\varepsilon,\delta)$ as ours. Beating it means matching its non-private accuracy within about two points on UCF-101. It also means exceeding the privatized version by a clear margin, more than three times the seed standard deviation, on SSV2 at $\varepsilon \le 8$.

The key ablation is a time-free marginal. It uses the same pipeline with $K=1$, so each class is stored as a single noisy mean and covariance. This isolates whether the temporal shape of the stored average is what matters. A second ablation shuffles slice order during replay. If shuffling barely hurts SSV2 accuracy, the temporal statistics are not actually being used.

We also include an SGP-style symbolic baseline. It stores per-video caption sequences and replays them through the text encoder. This tests whether language alone carries enough motion information.

Why not just…?

Why not just keep ESSENTIAL's 1–2 frame features per video, since features already look non-invertible? This is the main threat. Tasai et al. rely on exactly this argument. The argument fails if features can be inverted or if membership in the training set can be detected. We will run a feature-to-frame inversion decoder trained on public video, and a membership-inference attack, against ESSENTIAL's stored features and against our released statistics. Giving per-feature storage a formal guarantee requires noise on every single vector, with no averaging to dilute it. The prediction is that it collapses at budgets where our sums survive. If the attacks fail against plain features and noised ESSENTIAL holds up, our privacy argument weakens. We would then report that honestly.

Why not just store one prototype mean per class? On UCF-101 this may work about as well as our method, because appearance identifies most of its actions. This alternative is exactly our $K=1$ ablation. SSV2 settles it. Its classes include direction-reversed pairs, such as pushing something left versus right, that share a time-averaged appearance. If the time-free marginal matches us there, temporal slicing is unnecessary and we would drop it.

Why not just train a generative video model on each stage and replay its samples? A generator trained on a stage's data memorizes individual clips unless it is itself trained with differential privacy, which is expensive and unstable for video. CoMem and SGP both report off-manifold drift and loss of relational structure from unconstrained generation. On VQACL, CoMem's graph-conditioned replay reaches 55.8% accuracy versus 52.0% for the synthetic-replay method GIFT. Our sampler is a generator too, but its entire parameter set is the privatized released statistics. The privacy cost is therefore paid once and is fully accounted.

Why not just caption every clip and replay text? Captions lose fine physical detail, as noted by Optimus-1. Per-video captions are still per-instance records that can leak. The symbolic baseline measures the accuracy side of this, and the membership-inference attack applies to it unchanged.

Proposed Experiments

We use class-incremental splits of UCF-101, HMDB51, SSV2, and Kinetics-400, following the protocols ESSENTIAL evaluates. The metrics are:

  • average incremental accuracy;
  • average forgetting;
  • stored bytes;
  • the formal per-video $(\varepsilon,\delta)$;
  • empirical leakage, measured as membership-inference AUC and as inversion fidelity (feature-space similarity of reconstructed frames to the originals).

The main result is a curve of accuracy against $\varepsilon$, from roughly 1 to 16 plus the non-private case, for our method, privatized ESSENTIAL, and the time-free marginal, on each dataset.

Four checks decide the outcome:

  1. On SSV2, our method beats the time-free marginal at every budget, while the gap on UCF-101 is small. This shows that temporal structure is what is being preserved.
  2. Shuffling slice order erases most of the SSV2 advantage.
  3. At matched $\varepsilon$ and bytes, our method beats privatized ESSENTIAL.
  4. Membership-inference AUC against our released statistics stays near chance, while AUC against stored ESSENTIAL features does not.

If check 1 fails, the episodic-to-semantic temporal structure contributes nothing and the proposal reduces to private prototypes. If check 3 fails, aggregation offers no privacy advantage over noising individual features.