Abstract
This work was supported by the European Commission under the contract numbers FP7-ICT-/n610411 (MULTISENSOR) and H2020-RIA-645012 (KRISTINA).
Introduction
Punctuation serves as the essential "glue" of written language, encoding prosody, grammar, and syntax. In the field of Natural Language Processing (NLP), punctuation research has historically focused on restoration within speech transcripts—primarily periods and commas—relying heavily on acoustic cues like pause duration. However, tasks such as natural language generation (NLG) require a transition from raw, unpunctuated word sequences to readable, grammatically correct text.
Existing models often depend on language-specific features or complex syntactic trees. This research aims to provide a "function-neutral" and "language-neutral" system capable of handling the entire spectrum of punctuation (from semicolons to brackets) using only raw text as input.
Methodology
The core of the proposed system is a transition-based architecture, adapted from techniques commonly used in dependency parsing.
The Transition-Based Algorithm
The model operates using two primary data structures: an input buffer containing raw words and an output buffer containing words interleaved with generated punctuation. At each step, the model chooses between two actions:
- SHIFT: Moves the next word from the input buffer to the output buffer.
- GENERATE(p): Inserts a specific punctuation mark p after the current word in the output buffer.
Stack LSTMs and Character-Based Embeddings
To determine the optimal action, the model represents the current state using three Long Short-Term Memory (LSTM) units that track the input buffer, the output buffer, and the history of actions taken. This configuration allows the model to maintain a global context of the sentence.
Furthermore, the model utilizes bidirectional LSTMs to create embeddings from individual characters rather than whole words. This approach enables the system to handle rare words and capture morphological clues (such as prefixes or suffixes) that signal the need for specific punctuation marks.
Experimental Framework
The model was evaluated across five languages: Czech, English, French, German, and Spanish. Data was sourced from the CoNLL 2009 Shared Task treebanks. The study targeted the full range of punctuation found in these datasets, which varies significantly by language—for instance, Spanish includes inverted question marks (¿), while Czech includes section signs (§).
The neural model was compared against a "Look-ahead" baseline, which considers only a local 4-gram window (two words preceding and two words following the current position).
Results and Analysis
The LSTM-based model consistently outperformed the baseline across all metrics (Precision, Recall, and F-score) for all tested languages.
Key Findings
- Global Context: The Stack LSTM model's superiority over the look-ahead baseline demonstrates that punctuation is a global phenomenon requiring an understanding of the entire sentence structure.
- Linguistic Clues: Character-level representations successfully captured strong linguistic clues without the need for explicit part-of-speech tags or syntactic parsers.
- Cross-Lingual Robustness: The architecture performed well across languages with diverse punctuation philosophies, from the prosodic/rhetorical style of English to the strictly syntactic style of German.
- Punctuation Type: Periods were generally easier to predict (F-scores in the 80s and 90s) compared to commas (F-scores in the 50s to 70s), likely due to the higher frequency and subjective nature of comma usage.
Conclusion and Future Work
While the model showed high performance, results in French and Spanish were slightly lower, attributed to smaller training sets and longer average sentence lengths. Future research will explore cross-lingual models using multilingual word embeddings, potentially allowing models trained on high-resource languages to improve punctuation generation for low-resource languages. The system holds significant promise for integration into Automatic Speech Recognition (ASR) and grammatical error correction tools.
Summary updated