No article text was recorded for this version.
pre-layer normalization
Version 1 of 1113 papers149 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Transformers without Tears: Improving the Normalization of Self-Attention
- Blog: Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
- Blog: On Layer Normalization in the Transformer Architecture
- Blog: Transformers without Tears: Improving the Normalization of Self-Attention
- Blog: The NiuTrans Machine Translation Systems for WMT19
- Blog: An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets
- Blog: Critical Initialization of Wide and Deep Neural Networks using Partial Jacobians: General Theory and Applications
- Blog: TransMask: A Compact and Fast Speech Separation Model Based on Transformer
- Blog: DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
- Blog: NLCUnet: Single-Image Super-Resolution Network with Hairline Details
- Blog: Exploring Transformers for Large-Scale Speech Recognition
- Blog: Learning Deep Transformer Models for Machine Translation
- Blog: Lipschitz Constrained Parameter Initialization for Deep Transformers
- Blog: Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems
- Blog: Attention-Free Keyword Spotting
- Blog: Swin Transformer V2: Scaling Up Capacity and Resolution
- Blog: IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery
- Blog: The Path to Continuous Domain Adaptation Improvements by HW-TSC for the WMT23 Biomedical Translation Shared Task
- Blog: Impact of Layer Norm on Memorization and Generalization in Transformers
- Blog: Federated Learning with Dynamic Transformer for Text to Speech
- Blog: Identity Mappings in Deep Residual Networks
- Blog: Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
- Blog: Learning Deep Transformer Models for Machine Translation
- Blog: Emotion Recognition of Chinese Paintings at the Thirteenth National Exhibition of Fines Arts in China Based on Advanced Affective Computing
- Blog: Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation
- Blog: Arctic Sea Ice Image Super-Resolution Based on Multi-Scale Convolution and Dual-Gating Mechanism
- Blog: Understanding Scaling Laws via Neural Feature Learning Dynamics
- Blog: Why Deep Transformers are Difficult to Converge? From Computation Order to Lipschitz Restricted Parameter Initialization
- Blog: Very Deep Transformers for Neural Machine Translation
- Blog: Stabilizing Transformers for Reinforcement Learning
- Blog: Stabilizing Transformer-Based Action Sequence Generation For Q-Learning
- Blog: Impact of Layer Norm on Memorization and Generalization in Transformers
- Blog: Peri-LN: Revisiting Layer Normalization in the Transformer Architecture
- Blog: TAN without a burn: Scaling Laws of DP-SGD
- Blog: Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
- Blog: NVIDIA NeMo's Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21
- Blog: SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization
- Blog: A novel Deep Structure U-Net for Sea-Land Segmentation in Remote Sensing Images
- Blog: Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
- Blog: Scaled ReLU Matters for Training Vision Transformers
- Blog: Swin Transformer V2和特征融合的U-Net图像去噪方法
- Blog: On Layer Normalization in the Transformer Architecture
- Blog: Anatomy of Massive Activations and Attention Sinks
- Blog: A Survey of Transformers
- Blog: Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
- Blog: HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms
- Blog: What Language Model to Train if You Have One Million GPU Hours?
- Blog: Multi-Scale DenseNet-Based Electricity Theft Detection
- Blog: Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Blog: SPOC learner’s final grade prediction based on a novel sampling batch normalization embedded deep neural network method
- Blog: Scaling New Frontiers: Insights into Large Recommendation Models
- Blog: On the Residual Scaling of Looped Transformers: Stability and Transferability
- Blog: Character-based NMT with Transformer
- Blog: $\mathrmμ$nit Scaling: Simple and Scalable FP8 LLM Training
- Blog: Tanh Works Better with Asymmetry
- Blog: MonaCoBERT: Monotonic Attention Based ConvBERT for Knowledge Tracing
- Blog: Investigating Light-ResNet Architecture for Spoofing Detection Under Mismatched Conditions
- Blog: FlauBERT: Unsupervised Language Model Pre-training for French
- Blog: Transformer-based Acoustic Modeling for Hybrid Speech Recognition
- Blog: FlauBERT: Unsupervised Language Model Pre-training for French
Showing 60 of 113 papers.
History & citation
Current · Permanent link
- Normalization Placement for Stable Deep Transformers merged into this direction (50% overlap)
- Balancing Stability and Expressivity in Deep Transformers merged into this direction (73% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_pre_layer_normalization_51711e13,
title = {pre-layer normalization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/pre-layer-normalization/rdr_51711e13bc4240f7bf5e097d1ebaaddc},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}This exact version (version 1, frozen text):
@misc{lacuna_pre_layer_normalization_51711e13_v1,
title = {pre-layer normalization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/pre-layer-normalization/rdr_51711e13bc4240f7bf5e097d1ebaaddc/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}