Multidimensional Perceptron for Efficient and Explainable Long Text Classification

Yexiang Wang, Yating Zhang, Xiaozhong Liu, Changlong SunarXiv 2023 · DOI · arXiv

Abstract

Long document classification remains a challenge for standard Transformer models due to the quadratic complexity of self-attention. We present SWIPE (Segment-aWare multIdimensional PErceptron), a simple yet powerful framework designed to classify long documents efficiently while providing human-readable explanations. By replacing complex global attention mechanisms with a multidimensional perceptron and gated pooling strategies, SWIPE achieves state-of-the-art accuracy on long-text datasets. Crucially, it identifies specific "key segments" responsible for classifications without requiring manual rationale annotations, offering intrinsic explainability.

Introduction: The "Short-Sighted" Problem

Standard Transformer models like BERT face a significant bottleneck: their computational cost grows quadratically ($O(n^2)$) with text length, typically limiting them to a 512-token window. For long documents—such as legal contracts, medical records, or academic papers—researchers often split text into chunks and use Recurrent Neural Networks (RNNs) or additional attention layers to aggregate them.

However, these methods face two major hurdles:

  1. Efficiency: Aggregating many segments can still be computationally expensive.
  2. Explainability: In high-stakes fields like law and medicine, a simple label is insufficient; users need to know why a decision was made. While attention weights are often used as explanations, they can be misleading. Humans trust models more when they can point to a specific, continuous paragraph or sentence that triggered the decision.

Methodology

The SWIPE framework treats long-text classification as a search for the most "predictive" segments. The process follows three main steps:

  1. Truncation: Long text is split into segments based on fixed lengths, punctuation (sentences), or document structure (e.g., dialogue turns).
  2. Segment Encoding: Each segment is passed through a standard encoder (like BERT or RoBERTa) to generate a vector representation.
  3. The SWIPE Layer: Instead of a complex global attention mechanism, SWIPE applies a shared linear layer to every segment to calculate label scores. It then uses Gated Max/Sum Pooling to determine the final document label.
The SWIPE framework architecture, illustrating the transition from long text truncation to segment-level encoding and label distribution via gated pooling.

The core hypothesis is that the label of a long document is determined by its most significant segment. By using Max Pooling over segment-level scores, the model is mathematically forced to "pick" the segments that contribute most to the final classification. This makes the explanation (the chosen segment) an intrinsic part of the prediction process, rather than a post-hoc approximation.

Datasets

The model was evaluated on five diverse datasets to test its generality:

  • 20 NewsGroups: Multi-class classification of news articles.
  • IMDB & IMDB Long: Sentiment analysis of movie reviews (including a subset of reviews > 500 words).
  • arXiv: Multi-label classification of scientific papers (extremely long text).
  • MultiWOZ 2.2: Multi-label classification of dialogue turns.

Key Insights and Ideas

Efficiency through Linear Complexity

Unlike standard Transformers, SWIPE’s complexity scales linearly ($O(n)$) with document length. This makes it significantly more suitable for "extreme" long-text scenarios where quadratic models fail.

The "Sweet Spot" of Segment Length

The research identifies a critical trade-off between segment length and accuracy. If segments are too short, the model loses the local context needed for accurate encoding. Conversely, excessively long segments increase computational cost without necessarily improving performance.

Impact of segment length on model accuracy and training runtime, highlighting the 'sweet spot' for optimal performance.

For example, on the arXiv dataset, a segment length of approximately 152 tokens was found to be the "sweet spot" where accuracy plateaued and efficiency was maximized.

Unsupervised Explanations

SWIPE identifies "key sentences" in an unsupervised manner. It learns which segments are important simply by trying to predict the overall document label, eliminating the need for expensive human-labeled rationales.

Key Results

  • Superior Accuracy: SWIPE outperformed SOTA models like Longformer and CogLTX on the 20 NewsGroups and arXiv datasets. On arXiv, the RoBERTa + SWIPE variant achieved a Micro F1 of 82.47, surpassing the standard RoBERTa baseline of 79.86.
  • Sufficiency Test: To validate the explanations, the authors trained a separate classifier using only the segments SWIPE identified as important. This "explanation-only" model achieved nearly the same accuracy as the full-text model, proving the identified segments were truly the "smoking guns" of the classification.
  • Plug-and-Play Versatility: SWIPE improved the performance of every encoder it was paired with (BERT, RoBERTa, and ALBERT), demonstrating its utility as a versatile "head" for any backbone.

Limitations and Future Work

While effective, SWIPE has limitations. Simple truncation might split a vital sentence in half, suggesting a need for "intelligent" truncation that respects semantic boundaries. Additionally, the authors noted that encoding the entire text first before applying SWIPE leads to a loss of explainability, as self-attention "smears" information across all tokens. Maintaining the "bottleneck" between segments is crucial for clear, localized explanations.

Summary updated

Abstract

Because of the inevitable cost and complexity of transformer and pre-trained models, efficiency concerns are raised for long text classification. Meanwhile, in the highly sensitive domains, e.g., healthcare and legal long-text mining, potential model distrust, yet underrated and underexplored, may hatch vital apprehension. Existing methods generally segment the long text, encode each piece with the pre-trained model, and use attention or RNNs to obtain long text representation for classification. In this work, we propose a simple but effective model, Segment-aWare multIdimensional PErceptron (SWIPE), to replace attention/RNNs in the above framework. Unlike prior efforts, SWIPE can effectively learn the label of the entire text with supervised training, while perceive the labels of the segments and estimate their contributions to the long-text labeling in an unsupervised manner. As a general classifier, SWIPE can endorse different encoders, and it outperforms SOTA models in terms of classification accuracy and model efficiency. It is noteworthy that SWIPE achieves superior interpretability to transparentize long text classification results.
BibTeX
@misc{wang2023multidimensional,
  author = {Yexiang Wang and Yating Zhang and Xiaozhong Liu and Changlong Sun},
  title = {{Multidimensional Perceptron for Efficient and Explainable Long Text Classification}},
  year = {2023},
  eprint = {2304.01638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.01638},
}
Figure