Many sequence tasks give one label per sequence: an emotion for a whole utterance, a diagnosis for a whole ICU stay. A well-known training trick, target replication, copies that label onto every time step so the model is scored at every point in the sequence instead of only at the end. This gives far more gradient signal and teaches the model to commit to an answer early. The trick was developed for recurrent networks, which carry a running memory of everything seen so far. The faster, parallel alternatives, feedforward sequential memory networks (FSMNs, which replace the recurrent loop with a learned weighted sum over a fixed sliding window of neighbouring frames), can only see a bounded window. On a model like that, scoring step $t$ against the sequence label no longer asks "what have you concluded so far?" It asks "classify this short crop." That is ordinary random-crop training in disguise.
We propose to test this claim directly and then fix what it exposes. The fix is to give non-recurrent models a prefix-complete memory: a bank of exponentially forgetting accumulators, generalizing the ordinally-forgetting encodings used for feedforward language models. The bank is computed as a parallel prefix sum, so every supervised step sees the whole history without any sequential loop in training. We then ask a sharper question than "does dense supervision help FSMNs." Does it produce better anytime prediction, meaning accurate answers from partial sequences, than crop training, cumulative pooling, recurrent models, and causal attention, at a fraction of the recurrent training cost?
Core Claim
Dense per-step supervision is useful to a sequence model only to the extent that each supervised step can, in principle, see all the evidence the label depends on. For a bounded-window FSMN, target replication should reduce to crop-level training followed by averaging at test time. Its final accuracy will match crop training, and it will not produce the "confident early" behaviour that Learning to Diagnose with LSTM Recurrent Neural Networks observed for LSTMs.
Adding a learnable multi-rate decaying accumulator changes this. The accumulator has an infinite (though decaying) reach into the past, and it can be computed without recurrence at training time. With it, dense supervision becomes a genuine anytime objective. The prediction is that it should dominate crop training on accuracy-versus-fraction-observed curves, while keeping the throughput advantage that FSMN-family models hold over LSTMs.
Why this gap, and why the obvious version is not enough
The literature supplies each ingredient separately.
Two papers motivate the proposal. Emotion Identification from Raw Speech Signals Using DNNs showed that repeating the utterance label on every frame beats in-network utterance pooling on IEMOCAP. Its best system (70.6% weighted accuracy) still keeps an LSTM back-end. Pyramid Memory Block and Timestep Attention for Speech Emotion Recognition built a fully non-recurrent pyramid FSMN. That model gained 2.67% absolute unweighted accuracy over a CNN-biLSTM with roughly half the training time, but it was trained only with a single utterance-level loss after attention pooling.
The naive intersection is "put target replication on a pyramid FSMN." It is unstudied, but it runs straight into the problem above: with a bounded window, the replicated label at an early frame is often not inferable from that frame's input. Multitask Learning and Benchmarking with Clinical Time Series Data. and Lipton et al. avoided this problem because their LSTMs integrate the full prefix.
The closest existing fix comes from recommendation. Caser+ and CosRec+ made convolutional recommenders compatible with per-step supervision by using cumulative max-pooling and progressive average pooling over the prefix. The resulting CosRec+ beat SASRec by 53% NDCG@10 on Foursquare Tokyo. Those pooling operators are non-parametric and weight all history equally or by maximum, which blurs recent evidence on long sequences. They were tested only on discrete next-item targets, never on replicated static labels.
Acoustic Modeling with DFSMN-CTC and Joint CTC-CE Learning applied dense frame-level loss to deep FSMNs. There, frame targets come from external alignments and are genuinely local, so the tension we study does not arise.
The method-supplying work is The Fixed-Size Ordinally-Forgetting Encoding Method, which compresses a variable-length history into a fixed vector through a decaying forgetting factor. We reuse its forgetting encoding as a learnable, multi-rate, per-channel layer inside a DFSMN.
Language Models, Graph Searching, and Supervision Adulteration acts as a constraint. It shows that dense supervision can reward local shortcuts instead of the intended computation. We therefore design a diagnostic for exactly that failure rather than assuming that more supervision helps.
Research Program
The model is a deep FSMN with residual memory blocks, following Deep-FSMN for Large Vocabulary Continuous Speech Recognition. Its memory blocks are made causal (look-back only), so the step-$t$ output depends only on frames up to $t$. We add a prefix accumulator at chosen layers. For hidden features $z_\tau$ and $K$ learnable decay rates $\alpha_k \in (0,1)$, the accumulator is
$$s^{(k)}_t = \sum_{\tau \le t} \alpha_k^{\,t-\tau} z_\tau ,$$
normalized by $\sum_{\tau\le t}\alpha_k^{t-\tau}$ and concatenated with the local FIR memory output. This is a fixed linear filter with an infinite but decaying impulse response. It is computed with a parallel prefix scan (or an FFT convolution), so training has no step-by-step dependency. At deployment it streams in constant memory per step. Setting $\alpha_k = 1$ recovers Caser+-style progressive averaging, which is what makes that paper's operator a clean ablation.
Training uses Lipton-style target replication with an evidence schedule, to address the early-step label-noise problem that paper flagged:
$$\mathcal{L} = (1-\lambda)\,\frac{1}{T}\sum_{t=1}^{T} w_t\,\ell(\hat y_t, y) + \lambda\,\ell(\hat y_T, y).$$
Here $\ell$ is cross-entropy, $y$ the sequence label, $\hat y_t$ the step-$t$ prediction, $T$ the sequence length, $\lambda$ the weight on the terminal loss, and $w_t$ a non-decreasing weight on elapsed time. We compare a flat $w_t$ against a ramp over the first fraction of each sequence.
On MIMIC-III, decompensation is already a genuinely per-hour label, whereas phenotyping and in-hospital mortality are static labels that get replicated. The same code therefore covers both regimes.
The analysis has three parts. First, a receptive-field sweep: we vary the FSMN window $W$ relative to sequence length and measure the gap between target replication and crop training as $W/T$ grows. The claim predicts the two converge exactly when the prefix accumulator is absent. Second, we inspect the learned decay rates to see which timescales dense supervision recruits, compared with sparse supervision. Third, a shortcut diagnostic in the spirit of Frydenlund, to test whether per-step predictions track accumulated evidence or latch onto local cues. We shuffle the order of blocks of frames before step $t$ and measure how much $\hat y_t$ changes. A model that has learned a local cheat will be insensitive to shuffling; a model that accumulates evidence over time will not.
Baselines
The naive reference is the non-recurrent model trained the way Cao et al. trained it. A CNN–pyramid-FSMN with timestep attention pooling and one utterance-level loss, per Pyramid Memory Block and Timestep Attention, reimplemented on our splits. Beating it on final accuracy is necessary but not sufficient. The file gives only a relative figure for this model (2.67% absolute UA over CNN-biLSTM) and no absolute IEMOCAP score.
The ablation that isolates the key ingredient is the same causal DFSMN trained with target replication but without the accumulator. It is paired with a crop-trained twin: fixed-length crops matching the receptive field, the sequence label on each crop, and predictions averaged at test time. If the core claim is right, these two should be statistically indistinguishable. The proposal beats them if adding the accumulator improves area under the anytime-accuracy curve by a margin exceeding seed variance across five seeds.
The second ablation replaces learned multi-rate decay with the non-parametric prefix pooling of Caser+ and CosRec+. This is cumulative average and cumulative max, applied at the same layers with identical downstream heads. It tests whether learnable forgetting matters or whether any prefix summary suffices.
The strongest recurrent competitor is an LSTM trained with target replication or deep supervision. On MIMIC-III we follow the channel-wise multitask setup of Multitask Learning and Benchmarking with Clinical Time Series Data.. It reports in-hospital mortality AUC-ROC 0.870, decompensation AUC-ROC 0.911, phenotyping macro AUC-ROC 0.776, and length-of-stay kappa 0.451. On IEMOCAP, the reference is the TDNN-LSTM with attention and per-frame labels of Sarma et al., reporting 70.6% weighted accuracy on four classes with raw waveform input and augmentation. Protocols differ, so we rerun an LSTM with target replication on our own features rather than compare to that number directly.
Beating this baseline means matching the published MIMIC-III numbers within confidence intervals while showing a measured multi-fold training-throughput gain. Speech results suggest the room is there: roughly 3x in Deep-FSMN and about 7x in DFSMN-CTC. It also means an equal or better anytime curve.
The last baseline is a causal Transformer with the same per-step loss. It too sees the full prefix in parallel. All models get the same input features, the same equal hyperparameter-trial budget, and training to convergence under early stopping on the validation split. We report both wall-clock time and final metrics.
Why not just…?
Why not just use a causal Transformer with dense supervision? It already gives every step the full prefix and trains in parallel, so it is a real threat. Our argument is not that it will be less accurate. The accumulator model costs linear time in sequence length and streams at constant memory per step. That matters for long ICU stays at hourly resolution and for on-device audio, and attention offers neither property. Supervision adulteration in teacher-forced decoders (Frydenlund) is a reason to run the shuffle diagnostic on both models, not a reason to expect attention to fail here. If the Transformer matches accuracy at comparable cost on these sequence lengths, we will report that the contribution reduces to an efficiency result.
Why not just train a window classifier on random crops and average? Under our claim, this is what target replication on a bounded FSMN already is, which is why it is a baseline rather than a straw man. Crop training cannot produce a calibrated answer from "the first 20% of the stay" that improves as evidence arrives, because no crop-level model has an internal notion of how much history it has seen. The anytime curve is the experiment that separates the two. If crop training plus a running average of crop predictions matches the accumulator model on that curve, the accumulator adds nothing and we will say so.
Why not just use Caser+-style cumulative pooling? It is the simplest prefix-complete operator and has worked with dense supervision before. It weights all history uniformly or by maximum, however. In emotion and clinical deterioration, recent evidence often matters more than distant evidence, so we expect uniform weighting to hurt most on long sequences. The pooling-swap ablation, stratified by sequence length, settles this directly.
Why not just keep an LSTM with target replication? It works: it is the method of Lipton et al. and Harutyunyan et al. Its cost is sequential training. The proposal must show parity on accuracy and anytime behaviour with a measured speedup; accuracy gains alone would not justify the change.
Proposed Experiments
The first benchmark is the MIMIC-III clinical prediction suite (Harutyunyan et al.), using its four tasks and hourly resampling. We impute causally only, since that paper's retrospective preprocessing and Lipton's discussion both warn about back-filling future values. The file reports no result for any FSMN-family model on MIMIC-III, so these would be the first. The second benchmark is IEMOCAP, in both the four-class setting of Sarma et al. and the five-class setting of Cao et al. Both are reported on speaker-independent splits and use unweighted and weighted accuracy.
Metrics are the task-standard ones: AUC-ROC, macro AUC-ROC, and kappa on MIMIC-III; weighted and unweighted accuracy on IEMOCAP. Each is reported both at sequence end and as a curve over the fraction observed (10, 25, 50, 75, 100% of the utterance; hours 6, 12, 24, 48 for mortality). We also report training throughput and streaming latency per step.
Three results would decide the question. First, the equivalence test: without the accumulator, target replication and crop training should tie; if they do not, the core premise is wrong and the paper becomes an explanation of why. Second, the anytime test: with the accumulator, the area under the anytime curve should exceed both crop training and cumulative pooling, and approach or match the LSTM and causal Transformer. Third, the cost test: at matched final accuracy, the accumulator DFSMN should train several times faster than the LSTM. The receptive-field sweep and the shuffle diagnostic then explain why the curves separate. They show whether gains come from genuinely integrating evidence over time rather than from local shortcuts that dense supervision is known to reward.
References
- Learning to Diagnose with LSTM Recurrent Neural Networks Zachary Lipton
- Emotion Identification from Raw Speech Signals Using DNNs
- Pyramid Memory Block and Timestep Attention for Speech Emotion Recognition
- Multitask Learning and Benchmarking with Clinical Time Series Data
- Caser+ and CosRec+: Closing the Gap Between CNNs and Attention Models in SRS
- Acoustic Modeling with DFSMN-CTC and Joint CTC-CE Learning
- The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models
- Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More
- Deep-FSMN for Large Vocabulary Continuous Speech Recognition