NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents

Tamara Czinczoll, Christoph Hönes, Maximilian Schall, Gerard de MeloACL 2024 · DOI

Abstract

NextLevelBERT introduces a novel approach to processing long documents by reimagining Masked Language Modeling (MLM) at the level of "text chunks" (sentences or paragraphs) rather than individual tokens. By training a Transformer to reconstruct missing semantic embeddings from a frozen encoder, the model achieves a massive receptive field and high parameter efficiency. Evaluation on long-context benchmarks like BookSum and MuLD-Movie demonstrates that NextLevelBERT excels at capturing high-level narrative structures and long-range dependencies, outperforming significantly larger models on specific semantic tasks.

Introduction

The "Quadratic Bottleneck" remains a primary challenge in scaling Transformer architectures for long-document processing. Standard self-attention mechanisms scale quadratically ($O(n^2)$) with sequence length, making the ingestion of entire books or long scripts computationally prohibitive. While previous research has focused on sparse attention or linear approximations, NextLevelBERT shifts the focus to the input unit itself. By treating an entire sentence or paragraph as a single "token," the model can process hundreds of thousands of words within a standard Transformer window, enabling the capture of global narrative logic that word-level models often overlook.

Architecture and Methodology

NextLevelBERT utilizes a hierarchical architecture designed for semantic reconstruction.

The NextLevelBERT architecture: Text chunks are encoded into vectors, augmented with positional embeddings, and processed by a Transformer to predict masked chunk representations.

The process follows four key stages:

  1. Input Encoding: Documents are partitioned into chunks (e.g., 16–512 tokens).
  2. Frozen Semantic Encoder: Each chunk is mapped to a fixed-size vector using a pretrained sentence encoder (e.g., MiniLM).
  3. Next-Level MLM: A Transformer processes these vectors. In a variation of the BERT recipe, 15% of chunk vectors are masked. The model is tasked with reconstructing the original $d$-dimensional embedding of the missing chunk.
  4. Optimization: The model uses a Smooth L1 loss for pretraining. Crucially, initializing NextLevelBERT weights with those of the underlying chunk encoder was found to be vital for convergence.

Experimental Evaluation

The model was pretrained on the Books3 dataset and evaluated across several long-context benchmarks, including BookSum (matching chapters to summaries), MuLD-Movie (character classification in scripts), and QuALITY (multiple-choice QA).

Mean Reciprocal Rank (MRR) on BookSum documents across different text lengths. NextLevelBERT maintains competitive performance even as document length increases, outperforming Longformer and matching larger models like Nomic.

Key Results

On the BookSum benchmark, NextLevelBERT (using 256-token chunks) achieved a 65.45 MRR, surpassing the much larger Nomic Embed (65.07) and significantly outperforming Longformer (1.92). In the MuLD-Movie task, the model reached an 85.70 F1 score, demonstrating a superior ability to track character arcs across extensive scripts. However, performance on QuALITY was more modest (~32% accuracy), likely due to the dataset's requirement for fine-grained, token-level "needle-in-a-haystack" reasoning which the chunk-based compression naturally omits.

Discussion and Insights

NextLevelBERT's primary advantage is its massive receptive field. By using 512-token chunks, a standard Transformer can effectively model a context of over 260,000 tokens. This allows the model to learn "narrative logic"—predicting the semantic flow of scenes or arguments—rather than just local syntax. Furthermore, the model is highly efficient, achieving state-of-the-art results on specific tasks with only one-third of the parameters of its strongest competitors.

Conclusion and Future Work

While NextLevelBERT excels at "big picture" tasks, it faces a granularity trade-off, losing the ability to perform precise token-level reasoning. Future research may explore hybrid multi-scale models that combine high-level chunk representations with local token-level encoders to bridge the gap between global context and local detail. Currently optimized for English literature, extending this approach to technical domains and other languages remains a promising direction.

Summary updated

BibTeX
@inproceedings{czinczoll2024nextlevelbert,
  author = {Tamara Czinczoll and Christoph Hönes and Maximilian Schall and Gerard de Melo},
  title = {{NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.256},
}
Figure