Language models that read sensor data, such as electrocardiograms, brain waves or turbofan engine telemetry, are trained on large question-answer corpora. These corpora are mostly generated from templates applied to a much smaller number of recordings. ECG-QA, for example, turns 16,054 heart recordings into 267,539 training questions using 70 templates. Much of such a corpus is redundant. Worse, many of its questions can be answered from the wording alone, without looking at the signal. A model trained on this data can learn answer priors instead of learning to read waveforms. The literature also points to a specific weakness: questions that require relating several channels or several recordings to each other are where these models collapse. On ECG-QA's consecutive-comparison queries, the best model in ECG-QA drops to 20–22% exact match, against 74.6% on single verification questions.
We propose to curate these corpora with a small, cheap proxy model and a counterfactual test. The proxy scores each training example twice. The first score asks how much the answer's likelihood drops when the example's signal is swapped for another recording's signal; this measures signal reliance. The second score asks how much the likelihood drops when each channel is shifted in time by its own random offset. This shift keeps every channel's waveform intact but destroys the timing relationships between channels; it measures cross-channel coupling. We keep examples that are learnable, depend on the signal, and depend on the timing between channels. We discard examples answerable from text and examples that are never learned. We then train the full-size model on the curated subset. In two sentences: proxy-based data selection works because small models agree with large ones about which examples matter. For multichannel time series, the thing that matters is whether an example forces the model to read the signal and relate its channels, and a small model can measure that by corrupting its own inputs.
Core Claim
Our hypothesis is that, on multichannel time-series QA corpora, a subset chosen by proxy-measured signal reliance and cross-channel coupling will let a much larger target model match full-data accuracy at 25% of the data. We further expect this subset to beat simpler subsets of the same size. The two simpler subsets that matter are stratified sampling by template and recording, and filtering with a text-only "blind" model. We expect the gains to be largest on comparison and multi-channel questions, which are the ones current models fail.
Prior work leaves this question open in a specific way. Proxy selection has been shown for text and for static images. CoLoR-Filter scores sequences by the loss gap between a general prior model and a model fine-tuned on target data. S2L clusters a 70M model's loss trajectories and reaches full math fine-tuning performance with 11% of MathInstruct. Superfiltering shows that a 124M GPT-2's instruction-difficulty scores rank-correlate above 0.8 with larger models. XMAS (Naharas et al.) tracks singular values of a 0.5B vision-language proxy's cross-modal attention and prunes 50% of LLaVA-665k at no loss. These papers supply our method: a small proxy, a few checkpoints, cheap scoring. None of them asks whether an example's answer actually depends on the non-text input, and none has a notion of channels or time.
Our score is built differently. CoLoR-Filter contrasts two models on the same input. We contrast one model on true versus counterfactual inputs, which isolates the contribution of the signal. XMAS measures how stable attention is, which says nothing about whether attending to the signal was necessary. ARTIST is the closest temporal work. It selects informative segments inside a single series at test time with a reinforcement-learned controller, using 30–70% of each input. It is univariate, and it does not prune training data. Our selection operates over whole training examples, offline, before target training, and is explicitly multichannel.
Research Program
Let an example be a multichannel signal $x \in \mathbb{R}^{C \times T}$ (with $C$ channels and $T$ time steps), a question $q$ and an answer $a$. Let $\ell_k(x) = -\log p_{\theta_k}(a \mid q, x)$ be the proxy's answer loss at training checkpoint $k$. We define signal reliance and cross-channel coupling as
$$R_k = \mathbb{E}_{x' \sim \mathcal{S}(q)}\big[\ell_k(x')\big] - \ell_k(x), \qquad D_k = \mathbb{E}_{\tau}\big[\ell_k(\mathrm{shift}_\tau(x))\big] - \ell_k(x).$$
Here $x'$ is a signal drawn from another recording answering the same template, so the input stays in-distribution while its content changes. The term $\mathrm{shift}_\tau$ circularly shifts each channel $c$ by an independent random offset $\tau_c$. This preserves single-channel morphology and spectra but breaks synchrony across channels.
We compute both scores at roughly seven checkpoints of a one-epoch proxy run, matching XMAS's checkpoint count, using a few counterfactual draws per example. Following S2L's argument that trajectories carry more than single snapshots, we classify examples by the trajectories of $\ell_k$, $R_k$ and $D_k$. Text-answerable examples have low loss and near-zero reliance throughout. Never-learned examples keep high loss under every input and are likely label noise or beyond the proxy's reach. Signal-learned examples develop positive reliance as training proceeds. Within the signal-learned set, we sample evenly across trajectory clusters, as S2L does, and oversample examples with high coupling.
The proxy is OpenTSLM (Langer et al.) on its 270M Gemma backbone. That paper reports this backbone already outperforms GPT-4o on its tasks, so it is a credible proxy rather than a toy. The targets are an OpenTSLM-Flamingo model on Llama-3B and ITFormer-7B (Wang et al.).
We will also measure how well selection transfers from proxy to target. We will compute Spearman correlation of $R$ and $D$ between the proxy and each fully trained target on a held-out sample. The question is whether the >0.8 cross-scale agreement Superfiltering found for text holds for signal reliance.
A secondary question tests the multivariate direction directly. Many time-series language models process channels independently, a design used by AutoTimes (Liu et al.) that never attends across channels. We will train a channel-independent target alongside the cross-attention target. If high-coupling examples help only the cross-attention target, that is evidence the coupling score captures something real about cross-variable structure. We cannot exclude that it is merely a difficulty proxy unless the channel-independent target also fails to benefit.
Baselines
The naive reference points are training on the full corpus and training on a uniformly random subset of each budget (10%, 25%, 50%). Every subset method is run with the same target architecture, the same number of optimizer steps per budget, the same hyperparameter search (a fixed small grid shared by all methods), and three seeds. Proxy training and scoring costs are charged to our method in wall-clock accounting, as XMAS does when it reports a net 1.2× end-to-end speedup.
The full-data references are the published numbers. On EngineMT-QA, ITFormer-7B reports 65.07% accuracy on Perception and 88.69% on Reasoning (ITFormer). On OpenTSLM's chain-of-thought tasks, the reported F1 is 69.88% on Sleep-CoT and 67.64% on HAR-CoT (OpenTSLM). On ECG-QA, M3AE reaches 74.6% exact match on Single-Verify and 20–22% on Comparison-Consecutive-Query (ECG-QA).
We found no reported result on Lacuna for any data-selection method on time-series QA corpora. The nearest selection figure is from images: XMAS reports 95.4% average relative performance at a 10% budget on LLaVA-665k, against 94.1% for COINCIDE and 93.5% for random. That figure shows random subsets can be close on templated corpora, so random gets full seed treatment. Beating a baseline means a gap larger than two seed standard deviations on the per-task accuracy averaged across the four EngineMT-QA tasks and the ECG-QA question types.
Template-by-recording stratified sampling is the strongest cheap competitor. It allocates the budget evenly across the 70 ECG-QA templates (or the four EngineMT-QA task types) and across distinct recordings, which removes the obvious redundancy at zero cost.
A blind-model filter is the strongest conceptual competitor. It trains the same proxy with the signal input zeroed and drops examples the blind model answers confidently. This is CoLoR-Filter's prior/conditional contrast with "no signal" as the prior.
Ported dynamic selectors are the strongest existing methods. S2L is run on the proxy's answer-loss trajectories. XMAS is adapted by taking singular values of the proxy's gated cross-attention matrices, with the same checkpoint count and the same cluster sampling rule.
Two ablations isolate our ingredients. The first uses $R$ alone without the coupling term. The second uses single-checkpoint scores without trajectories.
Why not just…?
Why not just stratify by template and recording? This is a real threat. On a corpus generated from 70 templates, much of the redundancy is visible from metadata alone. Our bet is that signal reliance varies sharply within a template. Verification questions whose answer follows the base rate need no signal. Rare-finding questions need it badly. Stratification cannot tell these apart. The experiment settles it directly. If stratified sampling matches us within seed noise at the 25% budget, the counterfactual scores add nothing, and we would report that negative result. We will also report the within-template variance of $R$; if it is small, that alone predicts the outcome.
Why not just filter with a text-only blind model? This is the closest simple alternative, and it targets the same text-answerable examples. It has two weaknesses. First, a blind model sees an input distribution that never occurs at test time, whereas a signal swap keeps inputs realistic. Second, it says nothing about multichannel structure. The ablation using $R$ alone, compared against the blind filter, tests the first point. The coupling term, compared against both, tests the second. If the blind filter equals $R$-only selection, we keep it as the cheaper reliance measure, and the contribution narrows to the coupling score.
Why not just port S2L or XMAS unchanged? Both measure training dynamics: how an example's loss or attention evolves during training. Neither measures necessity of the input. A text-answerable example can have a perfectly stable, low loss trajectory, and S2L's even sampling across clusters will keep a share of such examples. XMAS deliberately prefers stable attention examples, which may be exactly the text-answerable ones. Running both under identical budgets, and reporting the fraction of each subset our proxy flags as text-answerable, will show whether this concern is real.
Why not just fix the architecture with cross-channel attention instead of curating data? The two are complementary rather than competing, and the channel-independent versus cross-attention comparison is designed to show how they interact.
Proposed Experiments
The main experiments use three corpora. On ECG-QA (267k training pairs, 12 leads) and EngineMT-QA (110k pairs, 32 channels), we train both targets at 10%, 25% and 50% budgets under every selector. We report per-question-type accuracy, separating single from comparison questions on ECG-QA and the four task types on EngineMT-QA. HAR-CoT and Sleep-CoT test whether the method carries over to chain-of-thought data, which has fewer channels and was generated from GPT-4o line-plot readings. On that data, the never-learned bin may also flag rationale artifacts.
Four checks decide whether the idea works. The first is whether our 25% subset matches full-data accuracy on both targets while stratified sampling and the blind filter fall measurably short. The second is whether the improvement concentrates on comparison and multi-channel questions, where full-data models score 20–22% on ECG-QA consecutive comparisons; uniform gains would suggest a generic difficulty effect instead. The third is whether a target trained on our subset shows a larger test-time loss increase under signal swaps and channel shifts than a full-data target. That would mean it reads the signal more, rather than just scoring higher. The fourth is whether proxy and target reliance scores rank-correlate strongly enough to justify the proxy. Weak correlation would mean signal reliance is scale-dependent, a negative result that constrains all proxy selection for sensor-language models.
References
- ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram
- CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
- Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories
- Adaptive Time Series Reasoning via Segment Selection
- OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data
- ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models