Code models are usually tested on puzzles whose specification is text: a docstring, an issue report, a failing test. A large class of real engineering code is specified differently. A vibration-monitoring engineer does not say "write a filter with cutoff 3.2 kHz." They hand over a recorded signal and a cleaned version and say "make this into that." The hard cases are signals whose frequency content drifts over time, such as a bearing under changing shaft speed or a transient impact buried in noise. We propose a benchmark of programming puzzles in which the specification is a set of input–output waveform pairs. The hidden program that produced the outputs is a signal-processing pipeline, and it is either time-invariant or time-varying. The model must write code that reproduces the transformation on held-out signals it has never seen.
The benchmark is built to answer one scientific question. Does continued pre-training of a code model on signal-processing code teach it something that general code training and tool access do not? And if it does, is that gain concentrated where the signal is non-stationary, meaning its statistics change over time? We make non-stationarity a controlled knob by pairing every time-varying puzzle with a stationary twin built from the same primitives. That turns a vague "domain helps" claim into a measurable interaction effect.
Core Claim
Our hypothesis concerns the "stationarity gap," defined as the pass rate on stationary twins minus the pass rate on their non-stationary counterparts. We expect general code models to show a large gap. We expect continued pre-training on a curated digital-signal-processing code corpus to shrink that gap more than an equal amount of extra training on generic code. We also expect it to shrink the gap more than giving a general model a sandbox in which to inspect the signals. If either compute-matched generic training or tool access closes the gap equally well, the claim is false. The benchmark would still stand as a contribution, but the pre-training story would not.
Why this intersection, and what already exists
The motivating evidence comes from three places. On the pre-training side, MonoCoder trained a 0.9B model from scratch on high-performance-computing C/C++ and reported that it beat larger general models on completion. A Comparative Study of PEFT Methods found that lightweight fine-tuning does not substitute for exposure to the domain during pre-training. Both papers evaluate only completion or text-to-code similarity, never execution against behavioral specifications. On the scaling side, the D-CPT Law fits continued-pre-training loss as a function of model size, tokens and domain mixture ratio. It reports that below roughly a 5% domain ratio, extra data can increase domain loss. This constrains our design: signal-processing code is a tiny slice of public code, so the mixture ratio must be swept rather than guessed.
On the signal side, adaptive decompositions such as empirical and variational mode decomposition exist because fixed Fourier analysis smears transients when frequency content drifts. Learned shrinkage networks like MLSCA-CW report 95.819% accuracy on the variable-speed University of Ottawa bearing data, falling to about 73.9% under mixed noise. The closest bridge between signals and language models is LLMs for prognostic analysis in mechanical fault diagnosis. It encodes wavelet images with a vision encoder and LoRA-tunes a language model to classify faults, reporting 98.72% on CWRU. That work, and APEX's domain-pretrained telemetry forecaster APEX, both end at a label or a forecast. Neither produces a program.
On the benchmark side, TraceFixer shows that runtime state helps repair: 63% top-1 on CodeNet, 13–20% over code-only inputs. However, it treats traces as discrete variable values from single-line bugs. Repository benchmarks such as SWE-bench, kGym and RepoFixEval specify tasks in text. Our proposal therefore differs from each strand. Unlike the signal–LLM work, the output is executable code scored on held-out signals. Unlike existing code benchmarks, the specification is behavioral and continuous. Unlike MonoCoder, domain pre-training is evaluated by execution, with a control that isolates domain content from extra compute.
We deliberately do not build a repository-repair benchmark driven by telemetry, even though the research file's gaps point that way. Such tasks would carry 17–30 minute virtual-machine evaluation loops like kGym's. Their noisy labels would also make an interaction effect undetectable at a feasible scale.
Research Program
Puzzle generator. We build a typed grammar of signal-processing primitives. It covers band-pass and notch filters, short-time Fourier masking, envelope extraction, soft-thresholding denoising (the operation learned in MLSCA-CW's shrinkage blocks), resampling by a speed profile (order tracking), mode decomposition followed by mode selection, and peak or onset detection.
Stationary and non-stationary twins. A puzzle is a sampled composition of these primitives. In the stationary twin, every parameter is fixed. In the non-stationary twin, one or more parameters follow a time-varying schedule tied to an observable in the input. For example, a notch frequency might track a shaft-speed chirp, or a threshold might track a drifting noise floor.
Carrier signals and what the model sees. Input signals are synthetic (chirps, impulse trains, mixed Gaussian/Laplacian/Brownian noise) and real (CWRU, University of Ottawa, and XJTU bearing recordings, all used in the cited papers). The model sees a short natural-language intent, such as "remove the speed-dependent tonal component." It also sees three to five input–output pairs, available as files in a Python sandbox with numpy and scipy.
Scoring. The model's program runs on ten held-out inputs. It passes if the normalized mean-squared error relative to the reference output falls below a per-primitive threshold, calibrated so the reference program with small parameter perturbations passes. Because scoring uses unseen signals, memorizing the example outputs fails.
Splits. Puzzles are generated freshly, which limits contamination. We hold out entire primitive compositions for an out-of-grammar test split. We also include about two hundred hand-written puzzles adapted from real condition-monitoring tasks, so the benchmark is not purely an artifact of our grammar.
Pre-training corpus and models. We mine public repositories that import scipy.signal, PyWavelets, librosa, or empirical/variational mode decomposition packages, then deduplicate them against benchmark reference code. Starting from an open code model at roughly 1B and 7B scale, we run continued pre-training at domain mixture ratios of 1, 3, 5, 10 and 25%. We fit the D-CPT functional form to held-out signal-processing code loss, and we test whether downstream pass rate tracks that loss.
Baselines
The naive reference point is an execution-free program. It returns the input unchanged or a least-squares linear filter fit to the example pairs. Every reported pass rate is shown alongside this floor, so a model that merely learns "output a smoothed input" earns no credit.
The strongest non-learned competitor is enumerative search over our own grammar with parameter fitting on the example pairs. On puzzles inside the grammar, this is close to an oracle. We report it as an upper reference rather than a competitor, and we evaluate claims about language models mainly on the out-of-grammar and hand-written splits, where search has no privileged access.
The strongest learned competitor is a frontier general model running as a sandboxed agent. It is the kind of tool-using setup that reaches 12.2% on SWE-bench Multimodal. It can compute spectrograms, inspect envelopes and iterate on its own code against the example pairs. It receives the same examples and the same per-puzzle budget of tool calls and executions as our models. No reported result found on Lacuna for program synthesis from waveform specifications. The nearest reported figures are GPT-4o's 11.25% patch pass rate on RepoFixEval and Claude 3.5 Sonnet's 54.15% syntax-error identification on RepoDebug. Both are different tasks and serve only to calibrate expectations.
Beating this baseline means one of two things. Either the domain-pretrained open model reaches a higher pass rate on non-stationary out-of-grammar puzzles at equal budget, or it shows a significantly smaller stationarity gap at comparable overall pass rate.
The key ablation is compute-matched continued pre-training on generic code from the same source distribution, with the same token count and schedule. Signal-processing content is the only difference, which separates domain content from additional training. A second ablation is LoRA fine-tuning of the base model on our puzzle training split. Following the CoNaLa comparison in Männistö et al., where CodeLlama 7B scored 0.277 CodeBLEU with LoRA against 0.295 with full fine-tuning, we expect this to help on in-distribution puzzles and little out of distribution. All learned variants use the same sandbox, sampling budget (pass@1 and pass@10), and hyperparameter search budget of a fixed number of trials per variant.
Why not just…?
Why not just give a frontier model a sandbox and let it compute spectrograms itself? This is the most serious threat. A capable agent can rediscover time–frequency analysis on demand. We do not assume it fails. The benchmark's matched twins measure exactly whether tool access alone closes the stationarity gap. If the agent's gap is as small as the domain-pretrained model's, we will report that domain pre-training is unnecessary for this skill. That would be a useful negative result, given how much effort goes into domain models like MonoCoder.
Why not just fine-tune on generated puzzles instead of pre-training? Puzzle fine-tuning teaches the grammar, and the grammar is ours, so in-grammar gains are expected and uninformative. The out-of-grammar and hand-written splits test whether the model has learned signal-processing judgment rather than our generator. The Männistö et al. finding predicts that fine-tuning will lag pre-training there, and our LoRA ablation tests that prediction directly.
Why not just serialize the signals as text, or summarize them with an FFT, and skip the sandbox? Thousands of samples per signal would bloat the context. RepoDebug shows accuracy already dropping from 51.48% to 43.07% between short and medium contexts. A global FFT summary also erases exactly the time-varying structure the non-stationary twins encode. We include a text-summary condition, giving the model global spectra and per-window statistics in the prompt, as a cheap baseline. We predict it will match the sandbox on stationary twins and collapse on non-stationary ones, which would itself confirm that the knob measures what we intend.
Decisive Checks
The central test is a two-way analysis of pass rate by model variant (domain CPT, generic CPT, LoRA, tool-using frontier agent) and by twin type (stationary, non-stationary) on the out-of-grammar split. The claim survives only if domain CPT shows a significant interaction: a smaller gap than generic CPT at the same compute. Significance is assessed by puzzle-level bootstrap over at least 1,000 twin pairs.
The second check concerns the mixture-ratio sweep. It should reproduce a D-CPT-style curve on held-out signal-processing loss. In particular, we look for whether the reported low-ratio inflection near 5% appears for this very small subdomain, and whether downstream pass rate moves with loss or diverges from it.
The third check asks whether the benchmark rewards analysis or pattern-matching. We measure performance on puzzles where the speed-tracking cue is removed from the natural-language intent. A model that truly reads the waveform should degrade little; one that keys on words like "variable speed" should collapse.
Any of three outcomes would falsify the pre-training hypothesis while leaving the benchmark useful: generic CPT matching domain CPT, the frontier agent closing the gap, or the gap vanishing once intent text is removed.
References
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- A Comparative Study of PEFT Methods for Python Code Generation
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
- Attention activation network for bearing fault diagnosis under various noise environments
- Large language models for prognostic analysis in mechanical fault diagnosis
- APEX: A Network-Native Time-Series Foundation Model for Forecasting and Anomaly Detection for Wireless Edge Operations
- TraceFixer: Execution Trace-Driven Program Repair
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution
- RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing
- SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
- RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models