Learned perception and control models in cars, robots, and drones run under hard deadlines. A correct answer that arrives late counts as a failure. When the processor is overloaded, today's best timing-assurance systems take the blunt route: they abort the overdue inference or skip the next sensor frame, and the controller is left acting on stale data. We propose a better fallback that still meets the deadline. One frozen network is cut into a short ladder of shallower configurations, and each configuration has its own worst-case execution time that can be bounded before deployment. Every shallower rung carries a small LoRA adapter, meaning a pair of thin matrices added to the frozen weights, trained to repair the accuracy lost by skipping layers. When a job is released, the scheduler picks the deepest rung that fits the remaining time. The system always produces a fresh output on time, and it stores only one base model.
The natural first idea at this intersection is "anytime LoRA": cut the adapter's rank when time runs short. We think that idea is a mistake, and explaining why motivates this proposal. Adapters make up almost none of the inference cost, so truncating them saves almost no time. They are useful here for a different reason. They are a cheap, low-memory way to make degraded configurations of a large network accurate. The timing savings come from the base network. LoRA supplies the accuracy recovery and the ability to store many operating points.
Core Claim
For a frozen weight matrix $W_0$ of size $d \times k$ with a rank-$r$ adapter $BA$, the extra multiply-accumulates per token per layer are
$$\frac{r(d+k)}{dk},$$
counted relative to the base matrix-vector product. For $d=k=4096$ and $r=8$ this is about 0.4%. Dropping the adapter entirely therefore cannot rescue a job that overruns its budget by more than a fraction of a percent. Rank-elastic schemes such as Flexi-LoRA choose a rank between 2 and 8 per input based on predicted difficulty. They report a 70.4% reduction in trainable parameters, which is a statement about parameters, not latency. They also add a routing step with variable cost.
Our claim has three parts. First, configurations with a fixed set of skipped layers give large, static, analyzable savings. Second, a per-rung repair adapter distilled from the full model recovers most of the lost accuracy. Third, choosing a rung once, at job release, from known slack keeps the whole system analyzable with ordinary mode-change schedulability analysis. What is new is the combination: degradation levels that each have certified worst-case bounds, are repaired by adapters, and share one base model. Prior work offers either certified but static networks or adaptive but unbounded ones.
Positioning
Two lines of work motivate the proposal. Designing Neural Networks for Real-Time Systems shows that neural inference can be given a proven worst-case execution time (WCET). It compiles Keras models to bounded, allocation-free C, runs them on the time-predictable Patmos processor with fixed-point arithmetic, and proves an 82.83 ms bound against a 100 ms F1/10 control deadline. It supports only one static network per binary. We reuse its code-generation discipline and extend it to a ladder of configurations that share weights.
Data-flow Availability / Kairos shows that timing violations in autonomous stacks can be detected and mitigated with little overhead, using abort, prioritize, or skip-next policies. It treats inference as a black box that either runs or is dropped. We replace dropping with a choice of rung.
Three LoRA papers constrain the design. The unified study of LoRA variants finds that well-tuned vanilla LoRA matches complex variants and has the lowest overhead. We therefore use plain $BA$ adapters and spend the hyperparameter budget on the learning rate. LoRA on the Go and Flexi-LoRA select or route adapters with input-dependent work at inference time. That is exactly the variable-time behaviour we exclude: our only runtime decision is a table lookup on slack. Compress then Serve shows that many adapters can share a jointly diagonalized basis while keeping above 99% Rouge-L. We borrow this to keep the rung-by-mode adapter set in fixed on-chip memory.
RTGPU supplies the GPU-side analysis. It provides persistent-thread partitioning at the streaming-multiprocessor level and response-time bounds that account for blocking by non-preemptive memory copies. We use it to bound adapter transfers whenever adapters cannot all stay resident.
Research Program
The ladder. Take a frozen base network $f$ with $L$ blocks and a task adapter $\theta^\star$ for the operating mode. For each rung $j$, fix a skip set $S_j$. The family is nested: $S_0 = \emptyset \subset S_1 \subset \dots \subset S_J$. For each rung, train a repair adapter $\theta_j$ on the blocks that remain:
$$\theta_j = \arg\min_\theta \; \mathbb{E}_x \, \ell\big(f_{S_j,\theta}(x),\, f_{\emptyset,\theta^\star}(x)\big) + \lambda\, \ell_{\text{task}}\big(f_{S_j,\theta}(x), y\big).$$
Here $f_{S,\theta}$ is the network with blocks $S$ skipped and adapter $\theta$ attached, $\ell$ is a distillation loss on outputs, and $\lambda$ weights the true task loss on labels $y$. The skip sets are chosen greedily on held-out data so that each step yields the largest WCET reduction per unit of accuracy lost.
The certified path. The ladder is emitted as bounded C in the style of Keras2C. Repair adapters are merged into the weights of the remaining blocks offline, once per rung. This works only when every rung's adapter fits in memory. Otherwise adapters stay unmerged at their 0.4%-level cost and their load is scheduled explicitly. Each rung gets a static WCET from platin on Patmos. This path is limited to small transformer policies, in the tens of millions of parameters.
The embedded GPU path. On Jetson-class hardware, a static bound is out of reach. There we use probabilistic WCET estimation in the spirit of Probabilistic WCET Estimation for Weakly Hard Real-Time Systems, together with RTGPU-style SM reservation. We state plainly that this yields probabilistic guarantees, not proofs.
The release-time policy. The selector picks the deepest rung $j$ whose bound, plus any adapter-load blocking, fits within the slack known at release. It never changes rung in the middle of a job. Because the choice is a lookup over a finite set of known WCETs, schedulability reduces to standard multi-mode analysis. We will additionally model the selector and the transfer engine in UPPAAL to check that no deadline is missed under the assumed arrival pattern of overload.
Baselines
The do-nothing reference runs the full model with its task adapter and no mitigation. Under overload it misses deadlines. It shows how bad the problem is and sets the accuracy ceiling.
The strongest existing system-level competitor is Kairos-style dropping, using abort and skip-next on the same task graph. Kairos reports a 64.4% collision reduction on Autoware and 83.3% on Jackal under simulated CPU overload, with 2.8% average overhead (Data-flow Availability). We will reimplement its policies on our stack, using the same overload traces and the same monitoring hooks. Beating it means fewer collisions or off-track events and lower freshness violations under identical traces, with comparable overhead.
The strongest certified-static competitor is a single small network compiled with the Keras2C discipline and sized to always meet the deadline. On F1/10 the reported bound is 82.83 ms against a 100 ms deadline (Designing Neural Networks for Real-Time Systems). We give it the same training data and the same total tuning budget as our ladder. We beat it only if our ladder has higher accuracy in nominal load and no more misses under overload.
The ablation that isolates the key ingredient is the identical ladder without repair adapters, which amounts to plain layer skipping. A second ablation is rank-only elasticity in the manner of Flexi-LoRA, which tests the claim that cutting adapter rank buys no meaningful time. A third replaces the per-rung repair adapters with separately fine-tuned full copies of each truncated network. It is matched for training compute and measures how much accuracy and memory the adapters trade away.
Why not just…?
Why not just pre-merge one adapter and certify one model? This is the right answer if the overload never happens, and our full-load rung is exactly that model. A single merged model has one fixed cost, so under overload its only fallback is dropping. Storing a merged full copy for every rung and every mode multiplies memory by the number of rungs times the number of modes. The full-copy ablation measures that memory cost directly against the adapter ladder.
Why not just deploy a small distilled network that always fits? This is the real threat. If a small model distilled from the full model matches the accuracy of our deep rungs in nominal conditions, the ladder adds complexity for nothing. The experiment settles it with matched training budgets. We will plot accuracy against WCET for the small model and for every rung. The ladder justifies itself only if its top rungs dominate the small model at nominal load while its bottom rung matches it under overload. We will report the outcome either way.
Why not just drop frames with Kairos? Kairos still let collisions through under overload, and it has no notion of a partial answer. Our closed-loop overload experiment compares the two directly on identical traces.
Why not just cut LoRA rank as anytime computation? The cost ratio in the Core Claim, $r(d+k)/(dk)$, says the savings are under one percent at typical sizes. The rank-only ablation will measure latency directly to confirm or refute this.
Proposed Experiments
Testbed one is the F1/10 LiDAR control task from the Keras2C paper. We upgrade it to a small transformer policy with track-condition modes and build a certified Patmos path. We also run it on a Jetson AGX Xavier with RT-Linux and ROS 2, following the 10 ms timed-loop setup of the F1TENTH real-time study. We inject CPU-overload traces and measure the deadline-miss rate, freshness and jitter as Kairos defines them, completion rate, and lap time on the Austria map. The reference there is the 16.79 s classical lap in the F1TENTH benchmark survey.
Testbed two asks whether repair adapters scale to reasoning-heavy perception. We build rung ladders for an 8B-class vision-language model and measure accuracy per rung on STSBench, where InternVL 2.5 8B reports 46.07% and GPT-4o with ground-truth 3D input reports 57.08%. We also measure the latency distribution of each rung on Jetson. This testbed makes no closed-loop safety claim.
The decisive checks are three. The first: across rungs, repair adapters must recover a clear majority of the accuracy that plain skipping loses. The second: under every overload trace, the ladder must incur zero misses on the certified path and must beat Kairos dropping on collisions or off-track events. The third: the small distilled model must not dominate the ladder's accuracy–WCET curve. If the third check fails, we will report that the ladder is unnecessary at this scale.
References
- Flexi-LoRA: Efficient LoRA Finetuning with Input-Adaptive Dynamic Ranks
- Designing Neural Networks for Real-Time Systems
- Data-flow Availability: Achieving Timing Assurance in Autonomous Systems
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
- LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
- Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
- RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks with Fine-Grain Utilization
- Probabilistic WCET Estimation for Weakly Hard Real-Time Systems
- UPPAAL—a Tool Suite for Automatic Verification of Real–Time Systems
- Optimizing Real-Time Performances for Timed-Loop Racing under F1TENTH
- Unifying F1TENTH Autonomous Racing: Survey, Methods and Benchmarks
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving