Ranking Safe-Policy Architectures by How Hard Their Safety Filter Works

Robots that work close to people increasingly run a neural network that proposes an action, followed by a small mathematical "safety filter" that minimally corrects that action so the robot never enters an unsafe state, such as coming too close to a person. The filter is usually a control barrier function (CBF): a function whose sign marks which states are safe, together with a rule that keeps the robot inside the safe set. Once the filter guarantees that the robot will not crash, the network's architecture no longer decides whether the robot is safe. It decides whether the robot is uselessly safe. A poorly shaped network produces commands the filter must override constantly, and the robot hesitates, oscillates, or freezes near people. Today nobody searches over these architectures. The networks are designed by hand, and checking any one candidate means full training plus closed-loop rollouts.

We propose to make architecture search for filtered safe policies affordable by ranking candidates on the filter's own behavior after a short training budget. The signals are how much the filter has to move the network's proposed action, and how sensitive the network's barrier parameters are to sensor noise. The claim is that these filter-derived quantities predict final closed-loop task completion far better than accuracy-oriented shortcut scores from architecture search, and noticeably better than short-training loss alone. That would make a search with human-in-the-loop safety constraints cheap enough to run on a handful of GPUs.

Core Claim

Neural architecture search (NAS) is the automated design of network structure. Its main cost is evaluating each candidate. Two families of methods cut that cost. Zero-cost proxies score untrained networks from a single forward or backward pass, and EZNAS evolves such scores to reach a Kendall Tau of 0.65 on NAS-Bench-201. Budgeted performance estimation shortens training in a principled way: Rethinking Performance Estimation in NAS picks the cheapest epochs and depth that preserve rank correlation and reports a 1,000x speedup over AmoebaNet. Both families rank architectures by classification accuracy. Neither has been asked to predict a control outcome, let alone the outcome of a network wrapped in a safety filter.

On the safety side, ABNet replaces the iterative quadratic-program filter of BarrierNet with a closed-form barrier layer and a multi-head design fused by a gate that provably preserves safety. It reports a 0% crash rate and a 100% passing rate on VISTA driving, where BarrierNet reached only about 33% passing because it froze near obstacles. That contrast is the motivating evidence. Both methods enforce safety through a barrier. What separated them was how often the robot could still make progress, and that property depends on how the network and the filter interact. ABNet's head count, head capacity, and backbone are all fixed by hand.

Our hypothesis is that freezing and oscillation come from two measurable properties of a partially trained network. The first is systematic disagreement between the nominal command and the safe set. The second is noise sensitivity in the barrier parameters that the network outputs, for example the gains in ABNet's adaptive penalty functions. We define the intervention ratio

$$I = \frac{\mathbb{E}_{x,\epsilon}\,\|u_{\text{safe}}(x+\epsilon) - u_{\text{nom}}(x+\epsilon)\|}{\mathbb{E}_{x,\epsilon}\,\|u_{\text{nom}}(x+\epsilon)\|}.$$

Here $x$ is a held-out observation, $\epsilon$ is injected sensor noise, $u_{\text{nom}}$ is the network's proposed action, and $u_{\text{safe}}$ is the filtered action. We pair $I$ with the average Jacobian norm of the predicted barrier parameters with respect to the input. Because ABNet's filter is closed form, both quantities cost one batched forward pass (plus one backward pass for the Jacobian) on logged states, with no rollouts. The proposed ranking score combines these two quantities with the short-budget imitation loss.

Research Program

The work has three parts.

The first part builds a search space and a ground-truth benchmark. The search space extends ABNet's design along the axes its authors fixed by hand: backbone depth and width, number of heads from one to eight, per-head capacity, and whether heads share early layers. It deliberately stays within families that keep the closed-form barrier layer and the provably safe gating, so every candidate stays safe by construction and only task progress varies. We sample about 200 architectures per task, train each fully, and evaluate each closed-loop. This produces the first tabular benchmark in which the target is filtered-policy task completion rather than accuracy. Building it is a real cost, but it is paid once and then shared.

The second part is the cheap-evaluation study. Following the budget-selection procedure of the budgeted-estimation paper, we train each candidate for a fraction of its epochs. At several budgets we record short-budget loss, the intervention ratio $I$, and the barrier-parameter sensitivity. We also compute standard zero-cost scores at initialization. We then measure how well each signal and each combination ranks the candidates against true closed-loop completion. We report both overall Kendall Tau and Tau within the top 10%, because EZNAS notes that zero-cost proxies break down among the best models, and that is exactly where a safety search must discriminate.

The third part is the human-proximity setting. We need a testbed where the danger is a person. In a two-link arm and a planar mobile-manipulator setting, we replay recorded human upper-body and hand trajectories from the multimodal human handover dataset (8,672 handovers from 26 people) as moving body segments near the robot. The barrier is the relative-velocity CBF of Zeng et al.. That CBF projects the relative velocity between the closest human and robot segments onto their common normal; Zeng et al. showed it avoids the stuck-in-crowd behavior of distance-only constraints and reaches targets 25% faster in simulation. A threat-aware barrier makes freezing depend on how human motion is predicted, which is precisely where architecture should matter. We implement the barriers with CBFkit.

We scope out monotone neural certificates. Monotone Neural Control Barrier Certificates makes verification cheap only for monotone dynamics and sign-constrained weights. Human-arm interaction is not monotone, so that line constrains the design rather than supplying it.

Baselines

The naive reference point is ABNet's own hand-designed configuration trained at full budget. If search cannot find architectures that beat it on closed-loop completion at matched parameter count and inference latency, the program has failed regardless of ranking correlations. Its reported result on VISTA is a 0% crash rate and 100% passing rate (ABNet paper). Because VISTA is saturated on passing, the headroom will come from the harder human-proximity and noisy settings. Wherever ABNet did not report a result, we report it ourselves; for the relative-velocity human-proximity setting there is no reported result on Lacuna for ABNet.

The strongest existing competitor for the ranking question is budgeted performance estimation that ranks by short-budget loss alone, run with the same epoch and depth budget selection as the original paper and the same compute as our method. It is the strongest because it is principled, cheap, and target-agnostic; its reported search cost is 0.33 GPU days with 2.66% CIFAR-10 error, but there is no reported result on Lacuna for it on control or safe-policy search. We beat it if, at equal training budget, our combined score raises top-10% Kendall Tau against closed-loop completion by at least 0.1 and yields a searched architecture with higher completion under noise.

The zero-cost reference is EZNAS-A, re-evolved on our benchmark with its original CPU budget, alongside Synflow and NASWOT. On NAS-Bench-201 CIFAR-10 these reach Kendall Tau of 0.65, 0.57, and 0.57 respectively (EZNAS paper).

The search-level baselines are random search and a grid over head count at equal GPU hours. We also include a safe-RL reference: Safety Gymnasium's PPO-Lagrangian on the hand-designed network, which it reports cut violations by 98% at the cost of a 45% drop in reward. This anchors the trade-off that a filtered, searched policy should improve on.

The key ablation removes the filter-derived signals from the score, leaving only short-budget loss. A second ablation removes noise injection when computing $I$. Together they isolate whether the filter's behavior under noise is the ingredient doing the work.

Why not just…?

Why not just use existing zero-cost proxies? They were built and validated to predict image-classification accuracy. The EZNAS authors themselves report that such proxies lose discrimination among the top 10% of models and assume repeated standard blocks. A multi-head network with a gated barrier layer is neither of those things. Our benchmark tests this directly by computing Synflow, NASWOT, and a re-evolved EZNAS on it. If they match our score, we will report that and drop our score.

Why not just rank by short-training loss with budgeted estimation? This is the real threat. A network that imitates the expert well early may simply be the one the filter rarely overrides. The loss-only ablation settles it. If the filter-derived signals add less than 0.05 top-10% Tau over loss alone, we will conclude that the filter adds no ranking information. The contribution would then shrink to the benchmark and the finding that ordinary budgeted estimation transfers to safe control, which is still worth reporting.

Why not just make ABNet bigger or tune the head count by grid? If completion depends mostly on capacity, a scaling sweep would find the answer without search. We include an equal-budget grid over head count and width, and report completion against parameter count for every sampled architecture. Search is only justified if good architectures differ from bad ones at the same size.

Why not just put a standard backbone behind an external filter, or use Lagrangian safe RL? The first is essentially BarrierNet's design, and ABNet's comparison shows it can freeze. Zeng et al. show that classical dynamic-window and potential-field methods get stuck in dense human zones. The second gives no invariance guarantee and, per Safety Gymnasium, pays heavily in reward. We keep both as reference points rather than competitors for the ranking question.

Proposed Experiments

We run three testbeds. The first is VISTA driving, where results can be compared to the published ABNet and BarrierNet numbers. The second is Safety-Gymnasium Goal and Button navigation with Point and Car robots, adapted by adding an explicit CBF filter, with performance reported in its normalized reward and cost. The third is the human-proximity replay task built from handover trajectories with the relative-velocity barrier. In every testbed, sensor noise is swept from 0% to 50% to match ABNet's stress test. Metrics are crash rate (expected near zero by construction, so any nonzero value flags a broken barrier), passing or completion rate, time to target, the fraction of time the robot is frozen, and search GPU-hours.

The decisive check is one number per testbed. After fully training about 200 sampled architectures, we compute top-10% Kendall Tau between each cheap score, at a budget of at most 10% of full training, and closed-loop completion under 30% noise. The proposal succeeds if the filter-derived score beats both loss-only budgeted estimation and every zero-cost proxy on at least two of the three testbeds. It must also produce, at a total search cost below the cost of fully training ten candidates, an architecture that exceeds hand-designed ABNet's completion in the human-proximity task at equal latency. A secondary check asks whether architectures ranked highly on VISTA keep their rank on the human-proximity task. That tells us whether a single searched design transfers across safety settings or whether each new human-interaction barrier needs its own cheap search.