Length-Aware Multi-Kernel Transformer for Long Document Classification

Guangzeng Han, Jack Tsao, Xiaolei HuangNAACL 2024

TL;DR

Long documents are a challenge for current NLP models, often leading to performance drops when document lengths vary. This paper introduces LAMKIT, a novel transformer architecture that uses multi-kernel encoding and length-aware vectorization to achieve robust and high-performance classification across a wide range of document lengths, outperforming existing SOTA models.

What Problem Does This Paper Address?

Neural language models, especially transformers like BERT, struggle with long documents due to their quadratic self-attention complexity, which limits input sequence length. Common solutions include segmenting documents into fixed-size chunks or using sparse attention mechanisms. However, these methods can lead to context fragmentation and a loss of crucial information, especially when dealing with documents of varying lengths. The paper highlights that existing SOTA models often overfit to specific document lengths and perform poorly on others, indicating a lack of generalizability. This is a significant problem because long documents are prevalent in many fields, such as legal and medical domains, where accurate classification is critical.

Methods

The proposed solution is the Length-Aware Multi-Kernel Transformer (LAMKIT), which addresses the challenges of context fragmentation and varying document lengths through three main modules:

  • Multi-kernel Encoding (MK): Inspired by TextCNN, this module uses multiple encoders with different kernel sizes to process text segments. This allows the model to capture diverse contextual representations from various perspectives, mitigating the context fragmentation caused by fixed-size segmentations. Each segment's representation is derived from the "[CLS]" token embedding of a pre-trained RoBERTa model.
  • Length-Aware Vectorization (LaV): This module explicitly incorporates length information into the model. It operates at two levels:
  • Segment Position Embedding: Encodes the position of text chunks within a document using a Transformer encoder, similar to positional encodings but applied to segments.
  • Length Vectors: These are derived from the MK outputs and further processed by a length encoder to capture document-level length features.
  • Hierarchical Integration: The segment vectors (from MK) and length vectors (from LaV) are combined and fed into a document encoder. The output is then processed through hierarchical pooling (max and average pooling) to generate length-aware document vectors, which are finally used for classification.

Datasets

The paper evaluates LAMKIT on five publicly available datasets:

  • Diabetes: Longitudinal patient records from the 2018 National NLP Clinical Challenges (n2c2).
  • MIMIC-III: A large, freely accessible critical care database.
  • ECtHR-A & ECtHR-B: Datasets containing legal case descriptions from the European Court of Human Rights.
  • SCOTUS: A collection of US Supreme Court opinions.

These datasets are noted for their varying document lengths, ranging from hundreds to tens of thousands of tokens, making them suitable for testing the model's robustness to length variations.

Key Insights and Ideas

  • Length Matters: The paper strongly emphasizes that document length is a critical factor influencing model performance, and existing SOTA models exhibit a significant lack of robustness across varying lengths.
  • Multi-Kernel for Context: Using multiple kernel sizes in the encoding process is crucial for capturing diverse contextual information and overcoming the limitations of fixed-size segmentations, which can lead to context fragmentation.
  • Explicit Length Encoding is Key: Directly incorporating length information through segment position embeddings and length vectors significantly enhances model generalizability and robustness on documents of different sizes.
  • LLMs Still Lag in Classification: A case study on ChatGPT reveals that while LLMs excel at generative tasks, they still underperform discriminative models like LAMKIT on long document classification tasks, suggesting that current LLM paradigms for classification need further development.

Key Results

LAMKIT consistently outperforms SOTA baselines across the five datasets, achieving an absolute improvement of up to 10.9% in F1 scores. Specifically:

  • LAMKIT shows an average improvement of 4.7% in F1-micro and 7.2% in F1-macro over the average performance of baselines.
  • Unlike many baselines that show performance drops on shorter or extremely long documents, LAMKIT demonstrates stable and robust performance across all document length quartiles.
  • Ablation studies reveal that removing either the Multi-Kernel (MK) mechanism or the Length-Aware Vectorization (LaV) module leads to significant performance drops, highlighting the importance of both components. Removing both modules results in the most substantial decrease in performance.

Limitations and Future Work

  • Applicability Beyond Classification: While LAMKIT is demonstrated for text classification, its applicability to other NLP tasks (like summarization or question answering) on long documents is a promising avenue for future exploration.
  • Hyperparameter Sensitivity: The paper mentions setting hyperparameters for optimal performance on validation sets, implying that further tuning might be beneficial.
  • Computational Resources: The authors note that limited computational resources influenced empirical choices, suggesting that more extensive hyperparameter searches or larger models could potentially yield further improvements.

Summary updated

BibTeX
@inproceedings{han2024length,
  author = {Guangzeng Han and Jack Tsao and Xiaolei Huang},
  title = {{Length-Aware Multi-Kernel Transformer for Long Document Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure