Multi-Granular Text Encoding for Self-Explaining Categorization

Zhiguo Wang, Yue Zhang, Mo Yu, Wei Zhang, Lin Pan, Linfeng Song, Kun Xu, Yousef El-Kurdi

Abstract

Deep learning models often function as "black boxes," which is problematic in high-stakes domains like medicine. This paper proposes a multi-granular text encoder that provides self-explaining classifications by highlighting evidence of varying lengths, from single words to long phrases. By organizing n-grams into hierarchical forest structures and utilizing tree-structured LSTMs, the model achieves superior accuracy and computational efficiency compared to standard CNNs and BiLSTMs.

Introduction

In fields such as clinical medicine, understanding the rationale behind a model's prediction is as critical as the prediction itself. While deep learning has advanced text categorization, existing "self-explaining" methods face several challenges. Many models only highlight single words, whereas medical meaning often resides in multi-word phrases (e.g., "petechial bleed"). Furthermore, sequential models like standard LSTMs often "smear" context across hidden states, making it difficult to pinpoint the exact source of evidence. CNN-based approaches, while better at capturing n-grams, often rely on fixed-length filters that lack the flexibility to capture natural language patterns efficiently.

Methodology

The authors introduce a Multi-Granular Text Encoder designed to extract intuitive, context-independent evidence. The core of this approach lies in the structural organization of n-grams.

Hierarchical Forest Structures

To avoid the computational redundancy of calculating every possible n-gram independently, the model organizes n-grams into hierarchical structures. The authors propose Left-branching and Right-branching forests, where shorter n-grams are reused to construct longer ones. This hierarchical reuse ensures that each sub-unit contributes uniquely to the representation, solving the "over-counting" problem found in simpler pyramid structures.

Tree-Structured LSTM and Attention

A tree-structured LSTM composition function computes node embeddings, with parameters shared across all n-gram orders. This design makes the model significantly more compact than a CNN. An attention mechanism is then applied over all nodes in the forest. The resulting attention weights ($\alpha$) directly indicate the importance of specific words or phrases, providing a transparent, human-readable explanation for the model's output.

Experimental Results

The model was evaluated on a public medical text classification dataset consisting of 14,102 medical abstracts categorized into five disease classes.

Performance and Efficiency

The BiForest model, which combines representations from both Left- and Right-branching forests, achieved an accuracy of 65.2%, outperforming BiLSTM (62.7%) and CNN (62.5%) baselines. Notably, the Forest models maintain higher accuracy as the n-gram order increases, whereas CNN performance tends to fluctuate.

Accuracy comparison between Forest, BiLSTM, and CNN models across different n-gram orders.

Beyond accuracy, the model demonstrates significant efficiency gains. The LeftForest was nearly 3x faster than BiLSTM and 2x faster than CNN during training for n-gram orders up to 7. It is also remarkably compact, requiring only ~168k parameters compared to ~848k for the CNN baseline.

Conclusion

The multi-granular text encoder provides a robust framework for interpretable text categorization. By leveraging hierarchical forest structures, it captures nuanced evidence while remaining computationally efficient. Future work may involve exploring extremely long-range dependencies and integrating these hierarchical structures with large-scale pre-trained models to further enhance performance.

Summary updated

BibTeX
@inproceedings{wang2019multi,
  author = {Zhiguo Wang and Yue Zhang and Mo Yu and Wei Zhang and Lin Pan and Linfeng Song and Kun Xu and Yousef El-Kurdi},
  title = {{Multi-Granular Text Encoding for Self-Explaining Categorization}},
  booktitle = {BlackboxNLP @ ACL},
  year = {2019},
  doi = {10.18653/v1/w19-4805},
}
Figure