Subword tokenization, the trick of repeatedly merging the most frequent adjacent symbol pairs until a vocabulary of reusable chunks emerges (byte pair encoding, or BPE), has recently moved from text to continuous signals. In time series, merging discretized values into variable-length "motifs" lets a model compress flat stretches heavily while keeping detail where the signal is busy. Label-efficient learning on medical signals has advanced in parallel. Models pretrained without labels on electrocardiograms (ECG) or electroencephalograms (EEG, scalp brain recordings) need only a small fraction of expert annotations afterward. The open question is what, if anything, the first idea contributes to the second. The obvious project is to build a multichannel BPE for ECG, train a token model, and report few-label accuracy. It runs straight into an awkward fact: a 3.8M-parameter continuous state-space model with no tokenizer at all dominates most clinical ECG categories.
We propose to split subword tokenization into its parts and keep only the part that helps when labels are scarce. Our hypothesis is that, for biosignals, the benefit of BPE-style tokenization in low-label fine-tuning comes mostly from where it places segment boundaries, and not from turning the signal into discrete symbols. If that holds, the right design keeps the raw continuous signal. It uses a multichannel BPE only to propose motif boundaries. Those boundaries then define variable-length patches and whole-motif masking for an ordinary continuous self-supervised model. We will test this with controls that separate boundary placement from variable length, sequence compression, and quantization.
Core Claim
In text, Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation simulated subword effects inside a byte-level model. It found that the gains come mainly from compression (more text processed per unit of compute) and from boundary information injected as an inductive bias, while extra vocabulary capacity matters little. No one has asked the same question for continuous signals or for few-label transfer. Our claim is that a continuous model given data-driven multichannel motif boundaries will match or beat fixed-patch continuous pretraining at 1–10% labels. It will also beat a fully discrete motif-token model on the same pretraining data and compute. A control with randomly placed boundaries of matched length will lose most of the gain. That pattern would show that boundary placement, not discreteness and not variable length as such, carries the benefit.
Research Program
The first component is a multichannel motif segmenter, which addresses the limitation stated in Byte Pair Encoding for Efficient Time Series Forecasting. That paper discretizes univariate series into equal-frequency bins, merges frequent neighbors into motifs, and reports strong zero-shot forecasting, but it never handles correlated channels and never evaluates few-label classification. We project each timestep of a 12-lead ECG, or of a fixed EEG montage, onto a few principal components learned from unlabeled data. We bin each component into a small number of equal-frequency levels, so that each timestep becomes one joint symbol drawn from a base alphabet of about 64 symbols. We then run standard BPE over these joint-symbol sequences. A merged token therefore spans all channels at once, which is what a pattern like ST elevation across leads II, III and aVF requires. The vocabulary is used only to read off segment boundaries. Throughout, we monitor how often each token appears in the labeled subsets, because Fishing for Magikarp showed that BPE vocabularies leave many tokens barely trained, and Unified Multimodal Understanding via Byte-Pair Visual Encoding found that large visual BPE vocabularies have many rarely activated merges. Keeping the model continuous is partly a defense against exactly this failure.
The second component is the boundary-aware continuous model. We encode the raw multichannel waveform inside each motif segment into one vector by pooling the output of a small convolutional stem over that variable-length span. A shared backbone then processes these segment vectors. For pretraining, we hide whole motifs and reconstruct their raw waveforms, or predict their latent codes contrastively. This differs from the fixed 6-second windows used by Brant and the fixed 1-second patches used by LaBraM: masking units follow signal structure instead of a clock. It also differs from the fixed-rate residual vector quantization in Hypnos because nothing is ever quantized at the model's input or target.
The third component is the decomposition study. Every model arm shares the backbone, the pretraining data, the number of pretraining tokens processed, and the fine-tuning protocol. The arms differ only in how the signal is cut into units and whether those units are discrete. The analysis asks which factor accounts for the low-label gap.
Baselines
The naive reference point is a purely supervised model trained on each label fraction from scratch. For ECG this is the S4 model and the Net1D convolutional network used as supervised references in Benchmarking ECG FMs: A Reality Check Across Clinical Tasks. These show how much any pretraining helps at all.
The strongest existing competitor is continuous contrastive predictive coding (CPC), which learns by predicting future latent features of the raw signal. Two papers support it. Self-supervised representation learning from 12-lead ECG data reports a PTB-XL macro AUC of 0.9418 for fine-tuned CPC, against 0.9318 for the best supervised model, and 0.927 for a frozen linear probe. The ECG benchmarking paper finds that ECG-CPC, a 3.8M-parameter state-space model, dominates 5 of 7 clinical categories. The same benchmark also reports that foundation models are 3.3x to 9x more label-efficient than supervised ones, and that ECG-JEPA leads below 1,000 labeled samples, so we include it at the smallest label budgets. We found no reported PTB-XL macro AUC at fixed 1%, 5% or 10% label fractions for either model in the file, so we will measure these ourselves.
For fairness, we re-pretrain CPC with fixed patches on exactly our unlabeled corpus, using the same backbone and parameter count. All arms get the same hyperparameter search budget, the same random label subsets, and the same five seeds. On EEG, the strongest reference is LaBraM, with reported TUEV balanced accuracy of 0.6616 and TUAB AUROC of 0.9162. We will run it, and our arms, at matched label fractions.
The discrete competitor is a multichannel extension of the time-series BPE tokenizer, trained by next-token prediction over the same vocabulary. This is the obvious idea, and it isolates the effect of quantization.
Two ablations isolate the key ingredient. The first, a random-boundary control, keeps the exact segment-length distribution and compression ratio of the BPE segmentation but places boundaries at random. The second, a boundary-indicator control, uses fixed patches but adds learned start and end markers wherever BPE boundaries fall, as in the byte-level simulation study. We count the proposal as beating the baselines only if it exceeds re-pretrained fixed-patch CPC by more than the seed-to-seed spread at 1% and 5% labels, on most task categories, in both domains, while the random-boundary control does not.
Why not just…?
Why not just use continuous CPC or a state-space model, which already wins? This is the main threat, and our design treats it as a hypothesis to test. The ECG benchmark gives clear evidence that continuous models with good inductive biases beat larger tokenized ones. Our proposal keeps everything continuous and changes only how the signal is cut and masked. Its comparison against fixed-patch CPC is therefore direct. If the two tie at low labels, we will report that motif boundaries add nothing beyond what CPC already captures.
Why not just use domain segmentation such as R-peak beat detection? For ECG this may work as well as BPE, and we include beat-aligned patches as an arm. If beat segmentation matches BPE boundaries on ECG, the value of the learned segmenter lies only in domains with no obvious unit. The EEG experiments, where no heartbeat-like anchor exists for events such as spike-and-wave discharges, would then decide whether the method is worth keeping.
Why not just apply a post-hoc rule like FLOTA? FLOTA improves text classification by keeping only the few longest tokens from an existing vocabulary. There is no existing vocabulary for multichannel ECG or EEG, so some unsupervised induction step is unavoidable. We do borrow its spirit: a "longest-motif-first" segmentation variant with a cap on the number of segments is one of our tuning options.
Why not just add augmentation and cross-subject alignment? Methods such as Euclidean alignment from Learning with Limited Labeled Electroencephalography Data address a different problem, the differences between subjects, and can be combined with any arm. We apply alignment identically to all EEG arms, so it cannot explain differences between them.
Proposed Experiments
We pretrain on unlabeled 12-lead ECG and evaluate on PTB-XL and a subset of the ECG benchmark's 26 tasks. The subset deliberately includes the structural targets, such as cardiac structure and function, on which current foundation models are reported to struggle. For EEG, we pretrain on pooled Temple University Hospital data and evaluate on TUAB and TUEV. Label fractions are 1%, 5%, 10% and 100%, sampled at the patient level, with linear probing and full fine-tuning reported separately. Metrics are macro AUC for ECG, balanced accuracy and AUROC for EEG, and expected calibration error, because overconfidence at 1% labels matters clinically. We also report minority-class AUC, since Self-supervised representation learning from 12-lead ECG data found the largest pretraining gains on rare conditions. Variable-length segments should help there if motifs isolate brief events.
Three checks are decisive. First, the boundary-aware continuous model must beat fixed-patch CPC at 1% labels, and the random-boundary control must recover less than half of that margin. Otherwise the gain is due to compression or variable length, not structure. Second, the discrete motif-token arm must fall behind the continuous boundary arm at low labels. We expect its deficit to be concentrated on fine-tuning examples dominated by tokens that are rare in the labeled set, which would link the glitch-token phenomenon to label efficiency outside text. Third, the boundary effect must appear on EEG as well as ECG, beyond what beat segmentation achieves. If all three hold, the field gets a usable result: take the segmentation from subword tokenization and drop the quantization. If they fail, we will have measured that tokenization's benefits in text do not carry over to label-scarce biosignal learning.
References
- Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
- Byte Pair Encoding for Efficient Time Series Forecasting
- Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
- Unified Multimodal Understanding via Byte-Pair Visual Encoding
- Brant: Foundation Model for Intracranial Neural Signal
- Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI
- Next-Token Prediction Enables Scalable Learning of Sleep Physiology
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Self-supervised representation learning from 12-lead ECG data
- An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers
- Emotion-Related Consciousness Detection in Patients With Disorders of Consciousness Through an EEG-Based BCI System