No article text was recorded for this version.
vocabulary size optimization
Version 1 of 1111 papers134 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law
- Blog: Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction
- Blog: Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
- Blog: Large Vocabulary Size Improves Large Language Models
- Blog: Optimizing the Size of Subword Vocabularies in Dialect Classification
- Blog: PETA: evaluating the impact of protein transfer learning with sub-word tokenization on downstream applications
- Blog: mALBERT: Is a Compact Multilingual BERT Model Still Worth It?
- Blog: Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
- Blog: PETA: Evaluating the Impact of Protein Transfer Learning with Sub-word Tokenization on Downstream Applications
- Blog: Fast Vocabulary Transfer for Language Model Compression
- Blog: Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for Maltese
- Blog: PETA: evaluating the impact of protein transfer learning with sub-word tokenization on downstream applications
- Blog: RIGA at SemEval-2022 Task 1: Scaling Recurrent Neural Networks for CODWOE Dictionary Modeling
- Blog: Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps
- Blog: DNAMotifTokenizer: Towards Biologically Informed Tokenization of Genomic Sequences
- Blog: Subword-augmented Embedding for Cloze Reading Comprehension
- Blog: Pixel-Level BPE for Auto-Regressive Image Generation
- Blog: CMU's Machine Translation System for IWSLT 2019
- Blog: Neural Machine Translation of Rare Words with Subword Units
- Blog: Language-Driven Region Pointer Advancement for Controllable Image Captioning
- Blog: CDA: a Cost Efficient Content-based Multilingual Web Document Aligner
- Blog: Tokenization is Sensitive to Language Variation
- Blog: Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
- Blog: MergeDistill: Merging Language Models using Pre-trained Distillation
- Blog: Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
- Blog: AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain
- Blog: EfficientFCN: Holistically-Guided Decoding for Semantic Segmentation
- Blog: Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces
- Blog: An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
- Blog: Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages
- Blog: Optimizing Segmentation Granularity for Neural Machine Translation
- Blog: Maximum a posteriori Based Decoding for CTC Acoustic Models
- Blog: Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations
- Blog: KLAPrompt: Infusing Semantic Knowledge into Pre-trained Language Models by Long-answer Prompt Learning
- Blog: Pretraining without wordpieces: learning over a vocabulary of millions of words
- Blog: From Bias to Balance: How Multilingual Dataset Composition Affects Tokenizer Performance Across Languages
- Blog: Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond
- Blog: Multitask Learning For Different Subword Segmentations In Neural Machine Translation
- Blog: An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
- Blog: AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain
- Blog: LLM Neologism: Emergence of Mutated Characters due to Byte Encoding
- Blog: Efficient Vocabulary Reduction for Small Language Models
- Blog: Multitask Learning For Different Subword Segmentations In Neural Machine Translation
- Blog: Fragment-based Pretraining and Finetuning on Molecular Graphs
- Blog: Dynamic Scene Classification Using Spatial and Temporal Cues
- Blog: Passport: Improving Automated Formal Verification Using Identifiers
- Blog: Sub-Word Similarity based Search for Embeddings: Inducing Rare-Word Embeddings for Word Similarity Tasks and Language Modelling
- Blog: SwahBERT: Language Model of Swahili
- Blog: Lerna: Transformer Architectures for Configuring Error Correction Tools for Short- and Long-Read Genome Sequencing
- Blog: Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition
- Blog: Acoustic-to-Phrase Models for Speech Recognition
- Blog: Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
- Blog: Discriminative Approach to Build Hybrid Vocabulary for Conversational Telephone Speech Recognition of Agglutinative Languages
- Blog: How Much is Enough? The Diminishing Returns of Tokenization Training Data
- Blog: LangBridge: Interpreting Image as a Combination of Language Embeddings
- Blog: AgentVocab: Structure-Aware Vocabulary Adaptation for Efficient LLM Agents
- Blog: Exploring Design Choices for Building Language-Specific LLMs
- Blog: De Novo Molecular Generation via Connection-aware Motif Mining
- Blog: Byte Pair Encoding is Suboptimal for Language Model Pretraining
- Blog: Fine-Tuning Transformers: Vocabulary Transfer
Showing 60 of 111 papers.
History & citation
Current · Permanent link
- Principled Vocabulary Selection and Scaling for Language Models merged into this direction (69% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_vocabulary_size_optimization_f83f66c8,
title = {vocabulary size optimization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/vocabulary-size-optimization/rdr_f83f66c879db453ca1d65327874c6607},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}This exact version (version 1, frozen text):
@misc{lacuna_vocabulary_size_optimization_f83f66c8_v1,
title = {vocabulary size optimization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/vocabulary-size-optimization/rdr_f83f66c879db453ca1d65327874c6607/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}