Abstract
Long document classification remains a challenge for standard Transformer models due to the quadratic complexity of self-attention. We present SWIPE (Segment-aWare multIdimensional PErceptron), a simple yet powerful framework designed to classify long documents efficiently while providing human-readable explanations. By replacing complex global attention mechanisms with a multidimensional perceptron and gated pooling strategies, SWIPE achieves state-of-the-art accuracy on long-text datasets. Crucially, it identifies specific "key segments" responsible for classifications without requiring manual rationale annotations, offering intrinsic explainability.
Introduction: The "Short-Sighted" Problem
Standard Transformer models like BERT face a significant bottleneck: their computational cost grows quadratically ($O(n^2)$) with text length, typically limiting them to a 512-token window. For long documents—such as legal contracts, medical records, or academic papers—researchers often split text into chunks and use Recurrent Neural Networks (RNNs) or additional attention layers to aggregate them.
However, these methods face two major hurdles:
- Efficiency: Aggregating many segments can still be computationally expensive.
- Explainability: In high-stakes fields like law and medicine, a simple label is insufficient; users need to know why a decision was made. While attention weights are often used as explanations, they can be misleading. Humans trust models more when they can point to a specific, continuous paragraph or sentence that triggered the decision.
Methodology
The SWIPE framework treats long-text classification as a search for the most "predictive" segments. The process follows three main steps:
- Truncation: Long text is split into segments based on fixed lengths, punctuation (sentences), or document structure (e.g., dialogue turns).
- Segment Encoding: Each segment is passed through a standard encoder (like BERT or RoBERTa) to generate a vector representation.
- The SWIPE Layer: Instead of a complex global attention mechanism, SWIPE applies a shared linear layer to every segment to calculate label scores. It then uses Gated Max/Sum Pooling to determine the final document label.
The core hypothesis is that the label of a long document is determined by its most significant segment. By using Max Pooling over segment-level scores, the model is mathematically forced to "pick" the segments that contribute most to the final classification. This makes the explanation (the chosen segment) an intrinsic part of the prediction process, rather than a post-hoc approximation.
Datasets
The model was evaluated on five diverse datasets to test its generality:
- 20 NewsGroups: Multi-class classification of news articles.
- IMDB & IMDB Long: Sentiment analysis of movie reviews (including a subset of reviews > 500 words).
- arXiv: Multi-label classification of scientific papers (extremely long text).
- MultiWOZ 2.2: Multi-label classification of dialogue turns.
Key Insights and Ideas
Efficiency through Linear Complexity
Unlike standard Transformers, SWIPE’s complexity scales linearly ($O(n)$) with document length. This makes it significantly more suitable for "extreme" long-text scenarios where quadratic models fail.
The "Sweet Spot" of Segment Length
The research identifies a critical trade-off between segment length and accuracy. If segments are too short, the model loses the local context needed for accurate encoding. Conversely, excessively long segments increase computational cost without necessarily improving performance.
For example, on the arXiv dataset, a segment length of approximately 152 tokens was found to be the "sweet spot" where accuracy plateaued and efficiency was maximized.
Unsupervised Explanations
SWIPE identifies "key sentences" in an unsupervised manner. It learns which segments are important simply by trying to predict the overall document label, eliminating the need for expensive human-labeled rationales.
Key Results
- Superior Accuracy: SWIPE outperformed SOTA models like Longformer and CogLTX on the 20 NewsGroups and arXiv datasets. On arXiv, the
RoBERTa + SWIPEvariant achieved a Micro F1 of 82.47, surpassing the standard RoBERTa baseline of 79.86. - Sufficiency Test: To validate the explanations, the authors trained a separate classifier using only the segments SWIPE identified as important. This "explanation-only" model achieved nearly the same accuracy as the full-text model, proving the identified segments were truly the "smoking guns" of the classification.
- Plug-and-Play Versatility: SWIPE improved the performance of every encoder it was paired with (BERT, RoBERTa, and ALBERT), demonstrating its utility as a versatile "head" for any backbone.
Limitations and Future Work
While effective, SWIPE has limitations. Simple truncation might split a vital sentence in half, suggesting a need for "intelligent" truncation that respects semantic boundaries. Additionally, the authors noted that encoding the entire text first before applying SWIPE leads to a loss of explainability, as self-attention "smears" information across all tokens. Maintaining the "bottleneck" between segments is crucial for clear, localized explanations.
Summary updated