Abstract
NextLevelBERT introduces a novel approach to processing long documents by reimagining Masked Language Modeling (MLM) at the level of "text chunks" (sentences or paragraphs) rather than individual tokens. By training a Transformer to reconstruct missing semantic embeddings from a frozen encoder, the model achieves a massive receptive field and high parameter efficiency. Evaluation on long-context benchmarks like BookSum and MuLD-Movie demonstrates that NextLevelBERT excels at capturing high-level narrative structures and long-range dependencies, outperforming significantly larger models on specific semantic tasks.
Introduction
The "Quadratic Bottleneck" remains a primary challenge in scaling Transformer architectures for long-document processing. Standard self-attention mechanisms scale quadratically ($O(n^2)$) with sequence length, making the ingestion of entire books or long scripts computationally prohibitive. While previous research has focused on sparse attention or linear approximations, NextLevelBERT shifts the focus to the input unit itself. By treating an entire sentence or paragraph as a single "token," the model can process hundreds of thousands of words within a standard Transformer window, enabling the capture of global narrative logic that word-level models often overlook.
Architecture and Methodology
NextLevelBERT utilizes a hierarchical architecture designed for semantic reconstruction.
The process follows four key stages:
- Input Encoding: Documents are partitioned into chunks (e.g., 16–512 tokens).
- Frozen Semantic Encoder: Each chunk is mapped to a fixed-size vector using a pretrained sentence encoder (e.g., MiniLM).
- Next-Level MLM: A Transformer processes these vectors. In a variation of the BERT recipe, 15% of chunk vectors are masked. The model is tasked with reconstructing the original $d$-dimensional embedding of the missing chunk.
- Optimization: The model uses a Smooth L1 loss for pretraining. Crucially, initializing NextLevelBERT weights with those of the underlying chunk encoder was found to be vital for convergence.
Experimental Evaluation
The model was pretrained on the Books3 dataset and evaluated across several long-context benchmarks, including BookSum (matching chapters to summaries), MuLD-Movie (character classification in scripts), and QuALITY (multiple-choice QA).
Key Results
On the BookSum benchmark, NextLevelBERT (using 256-token chunks) achieved a 65.45 MRR, surpassing the much larger Nomic Embed (65.07) and significantly outperforming Longformer (1.92). In the MuLD-Movie task, the model reached an 85.70 F1 score, demonstrating a superior ability to track character arcs across extensive scripts. However, performance on QuALITY was more modest (~32% accuracy), likely due to the dataset's requirement for fine-grained, token-level "needle-in-a-haystack" reasoning which the chunk-based compression naturally omits.
Discussion and Insights
NextLevelBERT's primary advantage is its massive receptive field. By using 512-token chunks, a standard Transformer can effectively model a context of over 260,000 tokens. This allows the model to learn "narrative logic"—predicting the semantic flow of scenes or arguments—rather than just local syntax. Furthermore, the model is highly efficient, achieving state-of-the-art results on specific tasks with only one-third of the parameters of its strongest competitors.
Conclusion and Future Work
While NextLevelBERT excels at "big picture" tasks, it faces a granularity trade-off, losing the ability to perform precise token-level reasoning. Future research may explore hybrid multi-scale models that combine high-level chunk representations with local token-level encoders to bridge the gap between global context and local detail. Currently optimized for English literature, extending this approach to technical domains and other languages remains a promising direction.
Summary updated