Processing One Dimensional Signals as Two Dimensional Images

864 concepts659 papers3 proposals

When dealing with one-dimensional sequential data—such as electrocardiograms (ECG), brainwaves (EEG), audio streams, or radio frequency sweeps—machine learning models face a fundamental challenge: raw time-series signals are often highly non-stationary, noisy, and compress complex spectral dynamics into a single changing amplitude over time.

To find a specific pattern in a raw 1D waveform, a neural network must learn to untangle phase, frequency, and temporal variations simultaneously. If a diagnostic feature depends on a specific frequency occurring at a specific time, a standard 1D convolutional or recurrent network must dedicate significant capacity just to learn these mathematical relationships from scratch.

The core insight of spectrogram-based feature extraction is to decouple these dimensions before the model ever sees the data. By transforming a 1D signal into a 2D time-frequency image, we map time to one axis, frequency to another, and signal intensity to pixel brightness or color. This transformation fundamentally re-frames a signal processing problem into a computer vision problem, unlocking the highly optimized spatial feature extraction capabilities of 2D Convolutional Neural Networks (CNNs) and Vision Transformers. As Rahul Sukthankar et al. observed in Semantic Learning for Audio Applications: A Computer Vision Approach, this shift allows researchers to leverage decades of computer vision breakthroughs directly on sequential data.

The Mathematical Bridges: From Waveforms to Visual Textures

To transform a continuous or discrete 1D waveform into a meaningful 2D image, models rely on mathematical operations that slice the signal across both time and frequency.

Short-Time Fourier Transform (STFT)

The most common bridge is the Short-Time Fourier Transform. Instead of taking the Fourier Transform of the entire signal (which loses all temporal information), the STFT slides a window across the waveform, computing the frequency spectrum within each localized time segment.

1D Waveform:  ───🪟───🪟───🪟───🪟─── (Sliding Window)
                 │     │     │     │
Fourier Trans:   ▼     ▼     ▼     ▼
2D Spectrogram: [█]   [█]   [█]   [█]  ──► Frequency Axis
                 └─────┴─────┴─────┴──► Time Axis

This creates a grid where each column represents a snapshot of the signal's frequency content. This technique is widely used across domains to expose features that are otherwise invisible in the raw temporal domain. For example, Tennison Liu et al. use STFT in Epileptic Seizure Classification with Symmetric and Hybrid Bilinear Models to analyze spectral power distributions over discrete intervals, while Peizhen Peng et al. utilize it in Seizure Prediction in EEG Signals Using STFT and Domain Adaptation to capture how brainwave frequencies evolve prior to a seizure.

Wavelet Transforms

While the STFT is powerful, it suffers from a fixed window size, creating a strict trade-off: wide windows give good frequency resolution but poor time resolution, while narrow windows do the opposite. To overcome this limitation, researchers frequently employ Wavelet Transforms. Wavelets dynamically adjust their scale, using long windows for low frequencies and short windows for high frequencies.

In Classify arrhythmia by using 2D spectral images and deep neural network, Tran Anh Vu et al. demonstrate that this multi-resolution approach captures the simultaneous relationship between time and frequency more effectively for non-stationary cardiac signals. Similarly, Yuxing Lin et al. leverage Frequency B-Spline wavelets in Wavelet transform and deep learning-based obstructive sleep apnea detection from single-lead ECG signals to generate high-resolution "scalograms" that expose subtle, irregular heart rhythm variations.

Why Computer Vision Architectures Excel on Spectrograms

Once a 1D signal is rendered as a 2D image, it inherits the geometric properties that make deep computer vision architectures so effective: translation invariance and local feature correlation. In a 2D time-frequency map, physical phenomena manifest as distinct visual geometries:

  • Horizontal Lines: Represent sustained, continuous frequencies or tones over time.
  • Vertical Lines: Represent sharp, instantaneous transients or broadband impulses (such as the R-peak of an ECG heartbeat or a sudden seismic shock).
  • Visual Textures: Represent background noise, static, or chaotic signal interference.

2D convolutional filters are explicitly designed to detect these types of spatial structures regardless of where they appear in the image. As Syed Muhammad Anwar et al. explain in Classification of Arrhythmia by Using Deep Learning with 2-D ECG Spectral Image Representation, applying spatial filters to these representations allows neural networks to extract dependent features that facilitate the separation of classes with highly similar raw temporal waveforms.

This visual abstraction eliminates the need for brittle, hand-crafted feature engineering. Sara Ross-Howe et al. demonstrate in Atrial Fibrillation Detection Using Deep Features and Convolutional Networks that treating heart rhythm classification purely as an image recognition task allows models to bypass manual noise filtering entirely. The network simply learns to recognize the chaotic visual "texture" of an irregular rhythm compared to a clean, repeating geometric pattern.

Furthermore, this representation allows models to separate biological signals from non-physiological artifacts. Marc Strik et al. show in Artificial intelligence for detection of ventricular oversensing... that converting voltage fluctuations into 2D maps allows networks to use the distinct visual texture of electrical noise to separate true cardiac arrhythmias from pacemaker interference.

Universality Across Domains

Because time-frequency imaging standardizes how sequential data is presented, the exact same underlying mechanism can be applied to wildly different fields of study.

Medical Diagnostics

In healthcare, mapping physiological signals to the 2D domain allows vision models to spot anomalies across various organs. Weijia Lu et al. combine unsupervised 2D feature extraction via Variational Autoencoders with recurrent layers in Method to Annotate Arrhythmias by Deep Network to automatically map cardiac electrical activity.

The approach scales from low-frequency heartbeats to high-frequency muscle movements. Ross Mandeville applies Audio Spectral Transformers (AST) to 2D spectrograms of transmembranous electromyography (EMG) signals in Deep learning enhanced transmembranous electromyography in the diagnosis of sleep apnea, capturing complex neuromuscular dependencies that are completely obscured in standard 1D waveform analysis.

Geophysics and Acoustics

Acoustic and seismic waveforms share identical physical properties with medical signals. In Recognizing Abnormal Heart Sounds Using Deep Learning, Jonathan Rubin et al. convert 1D phonocardiogram audio recordings into 2D Mel-frequency cepstral coefficient heat maps, mapping acoustic energy distributions to identify structural heart defects.

On a planetary scale, Lotfy H. Abdel Khaliq et al. apply this exact audio-centric logic to the earth's crust in REAVER: Real-time Earthquake Prediction with Attention-based Sliding-Window Spectrograms. By converting raw seismic amplitude data into Mel-Spectrograms, their model isolates the minute, instantaneous frequency shifts that signal the arrival of destructive P- and S-waves, providing early warnings in fractions of a second.

Radio Frequency and Telecommunications

Even invisible electromagnetic waves can be parsed visually. Wen Li demonstrates in Pattern-Aware Intelligent Anti-Jamming Communication: A Sequential Deep Reinforcement Learning Approach that representing radio frequency spectrum sweeps as visual waterfall diagrams or thermal charts allows deep networks to instantly categorize complex, adversarial jamming signatures.

Because these visual signatures are highly distinct, models can even collaborate securely without sharing raw data. Peng Wu et al. show in Jammer classification with Federated Learning that decentralized networks can successfully train global interference classifiers by processing local radio signals purely as 2D spectrogram images.

By transforming the abstract fluctuations of a 1D timeline into a structured 2D landscape of frequency and time, this paradigm allows deep learning models to see the true composition of a signal.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 626 papers
  2. Version 1 · 2026-10-04 · 959 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_spectrogram_based_feature_extraction_611a1303,
  title = {spectrogram-based feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spectrogram-based-feature-extraction/rdr_611a130369124b51aa666aa633d0a7a2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_spectrogram_based_feature_extraction_611a1303_v2,
  title = {spectrogram-based feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spectrogram-based-feature-extraction/rdr_611a130369124b51aa666aa633d0a7a2/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake