From syllable representation via gesture generation to articulation: A spiking neural model

Bernd J. KrögerarXiv 2026 · arXiv

This paper presents a biologically inspired spiking neural model that bridges the gap between abstract linguistic representations and the physical movements of speech. By leveraging the Neural Engineering Framework (NEF) and Semantic Pointer Architecture (SPA), the authors demonstrate how a hierarchy of neural populations can transform a sequence of syllables into the complex, overlapping "gestures" of the vocal tract required for articulation.

TL;DR

The researchers developed a spiking neural model that uses neural oscillators and ramp integrators to coordinate the timing and activation of speech gestures. This system successfully translates phonological syllable sequences into synchronized vocal-tract movements, providing a computationally explicit pathway from "thought" to "action" in speech production.

What Problem Does This Paper Address?

Speech production is a massive coordination challenge. When you say a word like "Schrank," your brain doesn't just send a single command. It must coordinate the tongue, lips, velum (for nasal sounds), and glottis (for voicing) in a precisely timed, overlapping dance.

Existing models often struggle to explain how static phonological forms (the words in our mental lexicon) are converted into real-time motor activity. Specifically, how does the brain manage "articulatory phonology"—the idea that speech is composed of overlapping gestures—within a framework of spiking neurons? This paper tackles the "sensorimotor" stage of this process: taking an ordered list of syllables and generating the continuous control signals that drive the muscles of the vocal tract.

Methods

The model is built using the Neural Engineering Framework (NEF), which provides a mathematical method for implementing high-level functional algorithms in populations of spiking neurons (specifically Leaky Integrate-and-Fire, or LIF, neurons).

The production pathway is organized into a functional hierarchy, as illustrated in Figure 1:

Functional parts of the production pathway, showing the transition from cognitive syllable sequences to articulatory movements.

Key technical components include:

  • Neural Oscillators: Each syllable occurrence is assigned a 2D neural oscillator. The "phase" of this oscillator acts as a local clock, providing a temporal coordinate for all gestures within that syllable.
  • Phase-Dependent Commands: As the oscillator rotates, it triggers specific "commands" at certain phase windows (e.g., "start lip closure at 54°").
  • Neural Ramp Integrators: These recurrent populations take the brief phase commands and shape them into smooth activation "ramps" with realistic rise and fall times. This separates the timing of a gesture from its kinetics.
  • Opponent Neural Populations: The final neural stage consists of pairs of populations representing opposing influences on an articulatory parameter (e.g., one for raising the tongue, one for lowering it). This mimics the agonist-antagonist organization of biological motor systems.
  • Geometric Articulatory Model: The decoded signals drive a geometric model (DYNARTmo) that visualizes the movement of the tongue, lips, and jaw.

Datasets

The authors evaluated the model using two types of data:

  1. Input Corpus: A set of 30 selections including eight lexical words (e.g., Kanne, Schrank, Tasse), isolated syllables, and a four-syllable nonsense sequence (bap–gum–dip–bap).
  2. Simulation Subcorpora: To test the robustness of the neural implementation, they ran 120 simulations using different "neural seeds" (random initializations of the neuron populations). This allowed them to assess how reliably the gestures were recruited across different "brains."

Key Insights and Ideas

  • Phase as a Coordinate: Instead of a global clock, the model uses syllable-specific oscillators. This allows for "syllable preparation time"—the next syllable's oscillator can start before the current one finishes, enabling the natural overlap (coarticulation) seen in human speech.
  • Decoupling Timing and Execution: By using separate populations for the oscillator (timing) and the ramps (activation), the model can independently adjust how fast a syllable is spoken versus how quickly the tongue moves to its target.
  • Disinhibition as a Trigger: The model uses a "disinhibition" mechanism where the cognitive controller releases the "brakes" on specific gesture pathways, allowing the oscillator's phase to reach the motor populations.
  • Reliability through Amplitude: The researchers found that "neural noise" or variability in small populations could cause gestures to fail to activate. They discovered that slightly increasing the command amplitudes (the "strength" of the signal) ensured 100% reliability across all tested neural realizations.

Key Results

  • Successful Articulation: The model generated coordinated articulatory realizations for all 30 items in the input corpus.
  • Reliability: In the paired simulation study, adjusting gesture command amplitudes increased the success rate from 57/60 to 60/60.
  • Temporal Overlap: The model successfully demonstrated inter-syllabic overlap. For example, in the word Kanne, the gestures for the second syllable (ne) began while the first syllable (ka) was still active, matching human speech patterns.
  • Resource Efficiency: The "execution core" for a four-syllable sequence requires approximately 54,400 neurons—a biologically plausible scale for these specific motor control functions.

Limitations and Future Work

  • Acoustics: The current model ends at geometric movement. The authors plan to integrate the DYNARTmo acoustic extension to generate actual speech sounds (waveforms) from these movements.
  • Sensory Feedback: The current model is "open-loop," meaning it doesn't listen to itself or feel the position of the tongue to correct errors. Integrating auditory and somatosensory feedback is a planned next step.
  • Learning: The "phasing rules" (when gestures start and stop) are currently hand-tuned based on linguistic theory. A major future goal is to have the model learn these coordination patterns from data.
  • Prosody: The model currently operates at a fixed nominal rate. Future versions will need to handle stress, intonation, and varying speech rates.

Summary updated

Abstract

Speech production requires a transformation from phonological representations into coordinated vocal-tract movements. This paper presents a spiking neural sensorimotor model within the Neural Engineering Framework and Semantic Pointer Architecture. Each syllable occurrence has a neural oscillator whose evolving phase coordinates its gestures and initiates the next syllable. Phase-dependent commands recruit separate neural populations for vocalic, oral-constriction, velopharyngeal, and glottal actions (gestures). Neural ramp integrators shape the rise and fall of gesture activation. Together with articulatory targets, these trajectories determine the extent and time course of articulatory movements. Opponent neural populations provide control signals to a geometric articulatory model. Thirty reference recordings cover an input corpus of eight lexical words, an alternative realization of one word, a syllable sequence, and isolated syllables. Two additional paired simulation subcorpora of sixty runs each, covering two words and one syllable sequence, assess activation reliability across neural realizations. Adjusting gesture command amplitudes increased successful realizations from 57/60 to 60/60 under the specified criteria, with ramp rate factors held constant. The model demonstrates how neural oscillator and ramp dynamics generate articulatory timing under specified phasing rules. It complements a preceding mental-lexicon model through an ordered syllable interface. Continuous online execution across all components and connection to the acoustic extension remain further integration steps. Accessible code and synchronized visualization support inspection of the neural activation trajectories and resulting vocal-tract movements.
BibTeX
@misc{kroger2026from,
  author = {Bernd J. Kröger},
  title = {{From syllable representation via gesture generation to articulation: A spiking neural model}},
  year = {2026},
  eprint = {2611.22499},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22499},
}
Figure