No article text was recorded for this version.
stochastic subword segmentation
Version 2 of 269 papers123 conceptsfrozen 2026-10-04
Papers in this version
- Blog: MaxMatch-Dropout: Subword Regularization for WordPiece
- Blog: BPE-Dropout: Simple and Effective Subword Regularization
- Blog: Distributional Properties of Subword Regularization
- Blog: Dynamic Acoustic Unit Augmentation with BPE-Dropout for Low-Resource End-to-End Speech Recognition
- Blog: StochasTok: Improving Fine-Grained Subword Understanding in LLMs
- Blog: Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
- Blog: Subword Regularization: An Analysis of Scalability and Generalization for End-to-End Automatic Speech Recognition
- Blog: Regularizing Word Segmentation by Creating Misspellings
- Blog: Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models
- Blog: Sailor: Open Language Models for South-East Asia
- Blog: Distributional Properties of Subword Regularization
- Blog: Tokenization Falling Short: The Curse of Tokenization
- Blog: How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?
- Blog: Optimizing Word Segmentation for Downstream Task
- Blog: Tokenization Falling Short: On Subword Robustness in Large Language Models
- Blog: The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task
- Blog: HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding
- Blog: The LMU Munich System for the WMT20 Very Low Resource Supervised MT Task
- Blog: Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- Blog: Subword Language Model for Query Auto-Completion
- Blog: EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
- Blog: Improving Self Consistency in LLMs through Probabilistic Tokenization
- Blog: Evaluating Robustness to Input Perturbations for Neural Machine Translation
- Blog: StochasTok: Improving Fine-Grained Subword Understanding in LLMs
- Blog: Jamo-Level Subword Tokenization in Low-Resource Korean Machine Translation
- Blog: Neural Machine Translation without Embeddings
- Blog: An Investigation of Phone-Based Subword Units for End-to-End Speech Recognition
- Blog: On the Robustness of Goal Oriented Dialogue Systems to Real-world Noise
- Blog: Proxy Compression for Language Modeling
- Blog: Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
- Blog: GRaMPa: Subword Regularisation by Skewing Uniform Segmentation Distributions with an Efficient Path-counting Markov Model
- Blog: Searching for Optimal Subword Tokenization in Cross-domain NER
- Blog: Evaluating Robustness to Input Perturbations for Neural Machine Translation
- Blog: Multi-view Subword Regularization
- Blog: Stochastic Tokenization with a Language Model for Neural Text Classification
- Blog: Joint Optimization of Tokenization and Downstream Model
- Blog: The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task
- Blog: Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond
- Blog: A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation
- Blog: Latent Sequence Decompositions
- Blog: NRC-CNRC Machine Translation Systems for the 2021 AmericasNLP Shared Task
- Blog: What do tokens know about their characters and how do they know it?
- Blog: What changes when you randomly choose BPE merge operations? Not much.
- Blog: SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
- Blog: You should evaluate your language model on marginal likelihood over tokenisations
- Blog: TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
- Blog: Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems
- Blog: Enhancing Long-form Text Generation Efficacy with Task-adaptive Tokenization
- Blog: Deep Unordered Composition Rivals Syntactic Methods for Text Classification
- Blog: HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding
- Blog: Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data
- Blog: Neural Machine Translation with Synchronous Latent Phrase Structure
- Blog: Forking Paths in Neural Text Generation
- Blog: Towards Character-Level Transformer NMT by Finetuning Subword Systems
- Blog: Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
- Blog: Beyond Characters: Subword-level Morpheme Segmentation
- Blog: Robust Dialogue State Tracking with Weak Supervision and Sparse Data
- Blog: Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification
- Blog: Robust Neural Machine Translation for Abugidas by Glyph Perturbation
- Blog: A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation
Showing 60 of 69 papers.
History & citation
Current · Permanent link
Cite this direction
Latest version (a living page):
@misc{lacuna_stochastic_subword_segmentation_28c3c3c5,
title = {stochastic subword segmentation},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/stochastic-subword-segmentation/rdr_28c3c3c5c7c34e1fbb67855d05c399b9},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}This exact version (version 2, frozen text):
@misc{lacuna_stochastic_subword_segmentation_28c3c3c5_v2,
title = {stochastic subword segmentation},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/stochastic-subword-segmentation/rdr_28c3c3c5c7c34e1fbb67855d05c399b9/v/2},
note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}