No article text was recorded for this version.
subword tokenization
Version 1 of 1372 papers455 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Scaffold-BPE: Enhancing Byte Pair Encoding with Simple and Effective Scaffold Token Removal
- Blog: The RWTH Aachen University English-German and German-English Unsupervised Neural Machine Translation Systems for WMT 2018
- Blog: Global Data Constraints: Ethical and Effectiveness Challenges in Large Language Model
- Blog: Neural Machine Translation with Byte-Level Subwords
- Blog: VEEF-Multi-LLM: Effective Vocabulary Expansion and Parameter Efficient Finetuning Towards Multilingual Large Language Models
- Blog: A Neural Network Architecture for Program Understanding Inspired by Human Behaviors
- Blog: Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
- Blog: Pointing to Subwords for Generating Function Names in Source Code
- Blog: MorphTok: Morphologically Grounded Tokenization for Indic languages
- Blog: Spelling Correction as a Foreign Language
- Blog: Neural Machine Translation for Extremely Low-Resource African Languages: A Case Study on Bambara
- Blog: The IIIT-H Gujarati-English Machine Translation System for WMT19
- Blog: Learning to represent programs with heterogeneous graphs
- Blog: Fast and Memory-Efficient Neural Code Completion
- Blog: TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models
- Blog: Neural Machine Translation for English-Tamil
- Blog: Effective Subword Segmentation for Text Comprehension
- Blog: NITS-CNLP Low-Resource Neural Machine Translation Systems of English-Manipuri Language Pair
- Blog: Definition Modelling for Appropriate Specificity
- Blog: Subword and Crossword Units for CTC Acoustic Models
- Blog: Sampling from Your Language Model One Byte at a Time
- Blog: WMT24 System Description for the MultiIndic22MT Shared Task on Manipuri Language
- Blog: Multi-turn Dialogue Response Generation with Autoregressive Transformer Models
- Blog: Better Generalization with Semantic IDs: A case study in Ranking for Recommendations
- Blog: Team-Risers@DravidianLangTech 2025: AI-Generated Product Review Detection in Dravidian Languages Using Transformer-Based Embeddings
- Blog: Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
- Blog: Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- Blog: CASICT Tibetan Word Segmentation System for MLWS2017
- Blog: Training a Tokenizer for Free with Private Federated Learning
- Blog: Subword-based Tagging by Conditional Random Fields for Chinese Word Segmentation
- Blog: A Survey of Machine Learning for Big Code and Naturalness
- Blog: The TALP-UPC Machine Translation Systems for WMT19 News Translation Task: Pivoting Techniques for Low Resource MT
- Blog: RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting
- Blog: Neural Machine Translation with Byte-Level Subwords
- Blog: A Survey of Deep Learning Techniques for Neural Machine Translation
- Blog: Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
- Blog: Using Neural Machine Translation Methods for Sign Language Translation
- Blog: VuLASTE: Long Sequence Model with Abstract Syntax Tree Embedding for Vulnerability Detection
- Blog: Transform and Tell: Entity-Aware News Image Captioning
- Blog: Analysis on Unsupervised Acquisition Process of Bilingual Vocabulary through Iterative Back-Translation
- Blog: Chinese-Portuguese Machine Translation: A Study on Building Parallel Corpora from Comparable Texts
- Blog: IRLAB-DAIICT@DravidianLangTech-EACL2021: Neural Machine Translation
- Blog: TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
- Blog: Automatic Speech Recognition Benchmark for Air-Traffic Communications
- Blog: Edinburgh Neural Machine Translation Systems for WMT 16
- Blog: Ayuuk-Spanish Neural Machine Translator
- Blog: SJTU-MTLAB's Submission to the WMT23 Word-Level Auto Completion Task
- Blog: Controllable Abstractive Summarization
- Blog: Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
- Blog: Transform and Tell: Entity-Aware News Image Captioning
- Blog: Spelling Correction as a Foreign Language
- Blog: Unknown Script: Impact of Script on Cross-Lingual Transfer
- Blog: Edinburgh's Statistical Machine Translation Systems for WMT16
- Blog: The RWTH Aachen University Machine Translation Systems for WMT 2019
- Blog: Neural Word Decomposition Models for Abusive Language Detection
- Blog: SubICap: Towards Subword-informed Image Captioning
- Blog: CVIT-MT Systems for WAT-2018
- Blog: From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
- Blog: NIT Rourkela Machine Translation(MT) System Submission to WAT 2022 for MultiIndicMT: An Indic Language Multilingual Shared Task
- Blog: Pre-trained Model-based Automated Software Vulnerability Repair: How Far are We?
Showing 60 of 372 papers.
History & citation
Current · Permanent link
- Subword Tokenization for Neural Code Intelligence merged into this direction (56% overlap)
- Representing Text through Subword Decomposition merged into this direction (78% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_subword_tokenization_32b8b48e,
title = {subword tokenization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/subword-tokenization/rdr_32b8b48e61714f97b3bfdfe59225fbbc},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 1, frozen text):
@misc{lacuna_subword_tokenization_32b8b48e_v1,
title = {subword tokenization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/subword-tokenization/rdr_32b8b48e61714f97b3bfdfe59225fbbc/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}