In federated learning, many phones each train a shared model on their own data and send back only the resulting update. On a smartphone, one local training round can take about half an hour (On-device Federated Learning with Flower). During that window the operating system may kill the training process to reclaim memory for a foreground app, the user may open something heavy, or the battery may run low. Current systems treat a killed client as a dropout. Its partial work is thrown away, the energy it spent is wasted, and the devices most likely to be killed are low-RAM phones, so their data is systematically underrepresented. This proposal borrows a technique from transactional databases running on persistent memory to make local training resumable at almost no cost. The trainer keeps its weights, optimizer state, and data-loader position in a memory-mapped file outside the app's managed heap. It alternates between two copies, so that one copy is always internally consistent. A killed client restarts at the last completed optimizer step instead of losing the round.
The key systems observation is that the common failure on phones is a process kill, not a power cut. Dirty pages of a shared file mapping belong to the kernel's page cache, not to the process, so they survive the process's death. If the training state already lives in such a mapping and every write is ordered so a torn update can always be detected, "checkpointing" no longer means serializing tensors and writing them out. It reduces to atomically updating one header word per step. The research question is whether this near-zero-cost durability turns federated rounds that would have been abandoned into useful contributions, and whether it does so better than simple periodic checkpointing.
Core Claim
Recovering from a kill costs two things. One is the price of saving state, $s$, measured in step-equivalents per save. The other is the work lost since the last save. With a save every $c$ steps and a kill probability $\lambda$ per step, the standard first-order analysis gives an expected waste per step of about
$$ W(c) \approx \frac{s}{c} + \frac{\lambda c}{2}, \qquad W^\ast = \sqrt{2 s \lambda} \text{ at } c^\ast = \sqrt{2s/\lambda}. $$
The first term is amortized save cost. The second is the average lost progress, which is half an interval. Conventional checkpointing (serialize the model and optimizer to a file) has a large $s$ on a phone, because it allocates on the managed heap, copies, and does flash I/O. That forces sparse checkpoints and leaves substantial waste. We claim that a double-buffered, off-heap, file-backed layout drives $s$ to roughly the cost of one header write, so $c=1$ becomes affordable and $W$ falls to nearly the cost of replaying one step. In federated terms, the fraction of started client rounds that deliver a usable update should rise sharply on memory-constrained devices, at a small per-step overhead. A second, more speculative claim is that file-backed state can be written back and reclaimed by the kernel more gracefully than anonymous heap memory. If so, the layout could itself lower the rate of low-memory kills. We treat that as a hypothesis to measure, not an assumption.
Research Program
The mechanism comes from DBXN (Reducing Transaction Processing Latency in Hardware Transactional Memory-based Database with Non-volatile Memory). DBXN separates fast volatile execution from durable state through parity versioning: odd versions mark work in flight and even versions mark completed persistence. That separation cut commit latency from about 145 ms to 22 μs. We do not need its hardware transactional memory or non-volatile memory, only its versioning discipline.
Our trainer keeps two slots, each holding a copy of the parameters and optimizer state, inside one memory-mapped file. A small header records the committed step number, the data cursor, and the random-number state. Step $t$ reads from slot $t \bmod 2$ and writes the optimizer's output into the other slot. Out-of-place updates are natural for most optimizers, so the second slot costs memory but no extra copy. After the step finishes, the header is updated with a single aligned write, and that write is the commit. A step torn by a kill leaves the header pointing at the intact slot.
The decision to place state outside the runtime heap follows the off-heap literature. On the JVM, managed heaps add per-object bloat and unpredictable collection pauses for machine learning workloads (MLlib). Raw byte layouts with metadata embedded in the buffer remove that overhead. Optimization Techniques for "Scaling Down" Hadoop shows this for primitive byte arrays. TurboStream's OSRBuffer (TurboStream) embeds status and length headers inside raw ring buffers and achieves zero GC interference. Our header-in-buffer design is a direct descendant of the OSRBuffer header idea. We do not need its lock-free multi-producer machinery, and we avoid its 15–20 second shutdown-start reconfiguration because nothing is reassigned on restart.
On the federated side, the resumable client plugs into Flower's Android and embedded clients (Flower). On the server it plugs into buffered asynchronous aggregation (FedBuff). FedBuff already accepts late updates and down-weights them by staleness, so it gives resumed clients a principled place to land. Synchronous FedAvg would simply time them out.
We will build three things. The first is the slot-and-header store for PyTorch on Jetson TX2 and for TFLite's trainable "base + head" models on Android. The second is a recovery path that remaps the file, validates the header, and resumes. The third is a policy that decides whether a resumed client should finish its round or restart from the newer global model, based on how many server versions have elapsed since it began.
FedHybrid (FedHybrid) is the closest existing work. It attacks the same memory wall on 4–12 GB phones by mixing recomputation and activation compression to avoid running out of memory. Our work is complementary rather than competing. FedHybrid lowers the probability of a kill and does nothing once one happens, while we lower the cost of a kill. FedHybrid also manages memory inside the ML framework and does not move state off the managed heap. We will run the two together.
Baselines
The naive reference point is the default behavior of Flower clients feeding a FedBuff server: a killed client is a dropout, and the server waits for other arrivals to fill its buffer. We use FedBuff's buffer size of $K=10$, which FedBuff reports as its best setting, and its staleness weighting. For calibration, FedBuff reports being up to 3.3× more efficient than synchronous FedAvgM and 2.5× more efficient than pure asynchronous FL (FedBuff). Our method beats this reference only if it reduces wall-clock time and client energy to target accuracy under the same kill process. Faster per-step training does not count.
The strongest direct competitor is periodic conventional checkpointing. The trainer calls the framework's own save routine every $c$ steps, writes to app storage, and reloads on restart. We will tune $c$ by grid search per device and kill rate, giving this baseline the same tuning budget as our method's single knob, which is whether to call msync at round boundaries for power-loss safety. Our method beats it if the measured total waste per step is lower at the baseline's best $c$, and if time-to-accuracy improves at matched energy. For checkpoint-resume of federated clients specifically, no reported result was found on Lacuna, so this baseline is measured fresh.
The third baseline is Flower's deadline cutoff: clients stop at a time limit $\tau$ and submit whatever they have. Flower reports this matches convergence with only a marginal 3% accuracy drop on CIFAR-10 (Flower). It recovers partial work from slow clients but not from killed ones, so it isolates whether our gains come from surviving kills or merely from accepting partial work.
Two ablations isolate the key ingredient. The first keeps the file-backed off-heap layout but saves only every $c$ steps by copying slots. This tests whether the double-buffer commit, rather than off-heap placement alone, is what drives $s$ toward zero. The second keeps per-step commits but resumes from the newer global model instead of finishing the stale round. This tests whether resuming the old trajectory is worth its staleness. FedHybrid will be run with and without our store to check that the benefits stack.
Why not just…?
Why not just checkpoint periodically with the framework's save function? This is the real threat, and the equation above shows exactly when it wins. If the measured $s$ for conventional saves on a phone is already small, say well under one step, then $W^\ast$ is already tiny and our design adds complexity for little gain. The proposal therefore begins by measuring $s$ for both approaches on Jetson TX2 and on mid-range Android phones across model sizes. If conventional saves turn out to be cheap, the contribution shrinks to the speculative memory-pressure effect, and we would say so.
Why not just drop killed clients and let asynchronous aggregation absorb it? FedBuff does tolerate missing clients, but it does not recover the energy they spent. Flower shows that local work on edge hardware is expensive: 1 to 10 epochs on a TX2 raised energy from 10 kJ to 100 kJ. Kills also fall disproportionately on low-memory devices. LEAF shows that federated performance at the 25th percentile of users can drop dramatically even when the median looks fine (LEAF). We will therefore report accuracy and participation for the device group most often killed, not just averages.
Why not just reduce memory so the process is never killed, as FedHybrid does? Memory reduction helps against kills caused by the trainer's own footprint. It cannot prevent kills triggered by the user opening a foreground app or by the OS deprioritizing background work, and FedHybrid itself needs a predictor because background usage fluctuates. Combining the two methods in the experiments shows how much residual kill rate remains after FedHybrid and how much of it our store recovers.
Why not just tune the garbage collector or request a larger heap? On phones the heap ceiling is bounded by physical RAM, and a larger heap makes the app a more attractive target for the low-memory killer. Collector tuning also does nothing for state lost at process death. Off-heap placement here is about survival across a kill, not about pause times.
Proposed Experiments
We will train ResNet-18 on non-IID CIFAR-10, matching Flower's TX2 setup, and use LEAF's FEMNIST and Shakespeare for naturally partitioned clients. All runs aggregate with FedBuff at $K=10$.
Kills are injected two ways. The first is a controlled Poisson process at several rates on TX2 devices. The second is real memory pressure on Android devices, created by scripted foreground apps, so that the actual low-memory killer chooses its victims. The first gives clean curves of $W$ against $\lambda$. The second tests ecological validity and the hypothesis that file-backed state is reclaimed more gracefully. We will also measure a power-loss variant, a hard device reset, to quantify what is lost without msync and what msync at round boundaries costs.
The metrics are per-step overhead, and replayed steps per kill. At the federated level they are the fraction of started rounds that yield an accepted update, wall-clock time and total client energy to target accuracy, and accuracy for the most-killed device group. We will also measure the extra flash writes caused by page writeback, since flash wear is a real cost.
The decisive checks are three. First, at the periodic baseline's best tuned interval, does our method show lower total waste and faster time-to-accuracy at matched energy? Second, does the double-buffer ablation show that per-step atomic commit, rather than mmap alone, accounts for the gap? Third, does the gain persist on top of FedHybrid? If the first check fails because conventional saves are already cheap, the idea in its current form is refuted.
References
- On-device Federated Learning with Flower
- Reducing Transaction Processing Latency in Hardware Transactional Memory-based Database with Non-volatile Memory
- MLlib: Machine Learning in Apache Spark.
- Optimization Techniques for "Scaling Down" Hadoop on Multi-Core, Shared-Memory Systems
- TurboStream: Towards Low-Latency Data Stream Processing
- Federated Learning with Buffered Asynchronous Aggregation
- FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
- LEAF: A Benchmark for Federated Settings