Voice assistants increasingly split work between a tiny always-on detector on the device and a larger keyword verifier running on a nearby edge server ("cloudlet"). When the user moves between base stations, or when an operator packs sessions onto fewer servers to save power, that verifier session has to move. The tools for moving it come from datacenter virtual-machine (VM) migration, where the energy cost is modeled as roughly proportional to the gigabytes of memory copied. A streaming speech model is a very different object. Its live state is a few kilobytes to megabytes of audio buffers and hidden activations, rewritten every 10 to 20 ms. It can also be rebuilt from scratch by replaying a short window of recent audio. We propose to measure and model what a handover of such a session actually costs in joules, and to train keyword spotting models so that their state is cheap to move.
The central bet is that for these small, fast-changing sessions, the fixed costs of starting and finishing a migration dominate, so volume-based models will mispredict energy. If that holds, the cheapest handover depends on the model architecture. It could be a full container migration, an explicit transfer of only the neural state, or an audio replay that recomputes the state on the destination server. A model whose state "washes out" within a bounded number of frames makes replay both cheap and accurate. Neither the migration literature nor the keyword spotting literature has looked at this choice.
Core Claim
Performance and energy modeling for live migration of virtual machines established that migration energy scales with total transferred data. Its prediction error stays under 10%, calibrated on enterprise workloads such as TPC-C, LINPACK and SPECweb2005 under Xen. Modelling energy consumption of network transfers and virtual machine migration splits migration into initiation, transfer and activation phases and finds Ethernet more efficient for small payloads. Both papers motivate our claim and constrain it. When the transferred volume collapses to kilobytes, the transfer phase should shrink toward nothing, and the initiation and activation phases should carry most of the energy. That second step is our hypothesis, not their finding.
Our claim has two parts. The first is a measurement claim: for streaming keyword spotting sessions, a phase-decomposed model that includes architecture-specific state size and recompute cost will predict per-handover energy much better than a volume-proportional model. The second is a design claim: models trained so their decoder state washes out within a known window $K$ can be handed over by replaying $K$ frames of audio. This beats both container migration and explicit state transfer on total energy, including the device's radio, while losing almost no detection accuracy around the handover.
Each handover option $o$ is scored by its predicted total energy:
$$E_o = E^{fix}_o + e_{net}\,B_o + E^{dev}_o + E^{comp}_o,$$
The four terms are defined as follows. $E^{fix}_o$ is the fixed cost of setting up and tearing down the move on the source and destination servers. $e_{net}$ is the network energy per byte. $B_o$ is the number of bytes moved. $E^{dev}_o$ is the device radio energy, which is non-zero only for replay. $E^{comp}_o$ is the destination compute spent recomputing state.
Research Program
The workload is a two-stage pipeline. A small on-device first stage, such as the depthwise separable network of Hello Edge, triggers a cloudlet-hosted second-stage verifier. We use three verifier families because they differ in how their state behaves.
The first family is a finite-receptive-field convolutional model in the style of MarbleNet, from MarbleNet. Its state is bounded by its receptive field. The second is a transducer, a decoder that carries a recurrent summary of everything decoded so far. We use the frame-skipping designs of TDT-KWS and MFA-KWS, whose history-dependent decoders are the hard case. The third is a streaming adaptation of Keyword Transformer, whose attention cache grows with context.
For each family we implement three handover paths. The first is three-layer container migration following Live Service Migration in Mobile Edge Clouds, which pre-caches the base and application layers and copies only the instance layer. Here the model weights are pre-cached as application layer. The second is explicit neural-state transfer, which serializes only the buffers, hidden states and caches as tensors. This path is also platform-neutral. That matters because the cited migration models are tied to homogeneous x86 hypervisors, while tensors can move from x86 to ARM edge nodes. The third is audio replay: the device re-sends the last $K$ frames and the destination recomputes the state.
The machine learning contribution is washout training. During training we randomly reset the decoder's recurrent state and caches mid-utterance. We then add a loss that penalizes divergence between the reset trajectory and the uninterrupted one after $K$ frames. This makes $K$ an explicit, tunable property. We sweep $K$ to trace the trade-off between detection accuracy and handover energy.
Baselines
The naive reference point is uncoordinated full container live migration, with the whole instance memory copied whenever the orchestrator decides to move the session. Live Service Migration in Mobile Edge Clouds is the closest system. It reports 3.7 s downtime for OpenCV face detection and 7.4 MB transferred for video streaming with layered containers. It did not measure energy and did not test speech workloads. We rerun it on our pipeline under the same CORE emulation it used, a 100 Mbps link with 25 ms latency.
The strongest existing method for predicting and choosing migrations is the volume-based model of Performance and energy modeling for live migration of virtual machines. It reports over 90% prediction accuracy and a 73.6% energy reduction against random migration selection on enterprise workloads. We refit its regression on our measured speech traces with the same number of calibration runs as our model. Beating it means two things. First, lower held-out per-handover energy prediction error on speech sessions. Second, lower realized energy when each model chooses the handover path.
The strongest keyword spotting reference is TDT-KWS, which reports 98.3% macro-recall on LibriKWS-20 against 82.5% for greedy RNN-T (TDT-KWS). Our washout-trained verifiers must stay within a pre-registered small margin of their uninterrupted recall to count as accurate. There is no reported result found on Lacuna for any migration energy measurement of a speech or keyword spotting workload, so the energy comparisons rest on our own measurements.
The key ablation is audio replay with an ordinary, non-washout-trained model at the same $K$. It isolates whether the training objective, rather than replay itself, keeps accuracy intact.
Why not just…?
Why not just run everything on the device and never migrate? For fixed small vocabularies this is the right answer, and we treat it as the floor for that regime. Accuracy falls as models shrink: res8-narrow keeps 90.1% on Speech Commands with 19.9K parameters, while KWT-3 reaches 98.6% on the V2 12-class task (Deep Residual Learning for Small-Footprint Keyword Spotting, Keyword Transformer). Open-vocabulary verifiers like MFA-KWS use 3.3M parameters. Our scope is therefore the second stage only. We report the first-stage-only operating point so readers can see when offloading is not worth it at all.
Why not just reconnect statelessly to the new server? It costs nothing to move, but the verifier loses history mid-utterance. That is exactly where transducers fail, because their decoders are history-dependent (MFA-KWS). We measure miss rate on keywords that straddle the handover instant to quantify this loss directly.
Why not just always replay a few seconds of audio? This is the real threat. A couple of seconds of 16 kHz audio is tens of kilobytes, possibly cheaper than any server-side migration. We do not argue around it; replay is one of our three paths. What the proposal adds is twofold. It shows how long the replay must be for each architecture without accuracy loss, which is what washout training controls. It also shows when the device's radio energy makes server-side state transfer cheaper. If replay with untrained models already matches washout-trained replay, the design claim fails, and the ablation is built to reveal that.
Why not just use the existing volume-based energy model? It may transfer fine. If fixed costs turn out small relative to transfer even at kilobyte scale, our measurement claim is false, and that is a reportable result. A head-to-head prediction error comparison on held-out speech handovers settles it.
Proposed Experiments
We build continuous streams by concatenating LibriKWS-20, Hey-Snips and MobvoiHotwords utterances with WHAM! noise at several signal-to-noise ratios. We then inject handovers at controlled positions: in silence, during speech, and mid-keyword. Mobility traces drive handovers across CORE-emulated base stations, and a separate consolidation scenario packs many concurrent sessions onto fewer hosts.
We measure energy with wall power meters on source and destination hosts, switch port counters converted to energy, and a device-side radio power trace. Detection metrics are miss rate and false alarms per hour, both overall and within a two-second window around each handover. We also record downtime and bytes moved.
Three checks decide the outcome. The first is whether the phase-decomposed model's held-out energy error is clearly below the volume-based model's on speech sessions while matching it on a TPC-C control. The second is whether washout-trained replay at the chosen $K$ reaches lower total energy than both container migration and state transfer while keeping within-window recall at the uninterrupted level. The third is whether the best path flips between convolutional and transducer models as link bandwidth and device radio cost vary, which would show that the choice really is architecture-dependent. A secondary check repeats state transfer from x86 to ARM hosts, where page-level migration is unavailable.
References
- Performance and energy modeling for live migration of virtual machines
- Modelling energy consumption of network transfers and virtual machine migration
- Hello Edge: Keyword Spotting on Microcontrollers
- MarbleNet: Deep 1D Time-Channel Separable Convolutional Neural Network for Voice Activity Detection
- TDT-KWS: Fast and Accurate Keyword Spotting Using Token-and-Duration Transducer
- MFA-KWS: Effective Keyword Spotting With Multi-Head Frame-Asynchronous Decoding
- Keyword Transformer: A Self-Attention Model for Keyword Spotting
- Live Service Migration in Mobile Edge Clouds
- Deep Residual Learning for Small-Footprint Keyword Spotting