LLMEmbed: Rethinking Lightweight LLM's Genuine Function in Text Classification

ChunLiu ChunLiu, Hongguang Zhang, Kainan Zhao, Xinghai Ju, Lin YangACL 2024 · DOI

Abstract

LLMEmbed introduces a paradigm shift in text classification, moving from slow, generative "prompt-based reasoning" to efficient "embedding-based transfer learning." By utilizing lightweight Large Language Models (LLMs) like LLaMA2-7B as frozen feature extractors rather than reasoning engines, the framework achieves state-of-the-art accuracy while requiring only 1.5% of the runtime and 1.8% of the electricity compared to sophisticated prompting methods.

Introduction

The prevailing trend in Natural Language Processing (NLP) is to treat classification tasks as generation problems. Researchers often employ "Chain-of-Thought" (CoT) or "Clue and Reasoning Prompting" (CARP) to guide LLMs through multi-step reasoning before outputting a label. However, this approach faces significant hurdles:

  1. Efficiency Bottlenecks: Serial reasoning is computationally expensive, leading to high latency and energy costs.
  2. Model Scale Dependency: Lightweight open-source models (e.g., 7B parameters) often struggle with hallucinations or fail to follow complex prompts compared to massive models like GPT-4.
  3. Prompt Engineering Overhead: High performance often depends on brittle, manually tuned prompts to align inputs with expected outputs.

LLMEmbed addresses these by asking whether the "knowledge" within a lightweight LLM can be harnessed more effectively without forcing it to act as a human-like reasoner.

Methodology

LLMEmbed treats the LLM as a sophisticated encoder within a transfer learning framework. Key technical components include:

  • Multi-Depth Feature Extraction: Rather than relying solely on the final hidden state, LLMEmbed extracts and averages token embeddings from multiple transformer blocks (specifically the last five layers of LLaMA2-7B) to capture a broader semantic spectrum.
  • Cross-Model Fusion: The framework concatenates embeddings from generative models (LLaMA2) with traditional discriminative models (BERT or RoBERTa).
  • Co-occurrence Pooling: A power-normalized co-occurrence pooling strategy aligns different embedding spaces into a unified representation, capturing high-order relationships between different embedding vectors.
  • Frozen Backbone: The LLM remains frozen, and only a lightweight classifier head is trained. This prevents "catastrophic forgetting" and ensures high-speed processing through parallelism.
LLMEmbed Architecture: The framework extracts embeddings from multiple LLaMA2 blocks and fuses them with BERT/RoBERTa features for classification.

Evaluation and Results

The authors evaluated LLMEmbed on five benchmarks: SST-2 and MR (sentiment analysis), AGNews (topic classification), and R8/R52 (news categorization).

Performance and Efficiency

LLMEmbed achieved an average accuracy of 96.20% on the LLaMA2-7B backbone, significantly outperforming the prompt-based CARP method (81.14%). Remarkably, this performance is comparable to GPT-3 (175B) using sophisticated prompting, despite LLMEmbed using only 4% of the parameters.

Main Results: Comparison of LLMEmbed variants against PLM and Prompt-based LLM methods across five classification benchmarks.

Beyond accuracy, the efficiency gains are transformative. Compared to prompt-based methods on the same backbone, LLMEmbed required just 1.5% of the runtime and 1.8% of the electricity. For a typical workload, the energy cost for LLMEmbed was approximately $0.025, compared to nearly $200 in API costs for a GPT-3 prompt-based approach.

Discussion

The results suggest that for lightweight models, the latent semantic space is more robust than the ability to generate coherent reasoning steps. The "genuine function" of a localized LLM in classification lies in its role as a high-dimensional feature extractor. Furthermore, the synergy between generative (LLaMA2) and discriminative (RoBERTa) models indicates they capture complementary semantic features.

Limitations and Future Work

While efficient, LLMEmbed currently faces challenges with very long-text classification without fine-tuning the backbone. Additionally, moving away from reasoning prompts reduces "human-readable" explainability. Future work aims to bridge this gap by combining the efficiency of embeddings with the interpretability of reasoning prompts.

Summary updated

BibTeX
@inproceedings{chunliu2024llmembed,
  author = {ChunLiu ChunLiu and Hongguang Zhang and Kainan Zhao and Xinghai Ju and Lin Yang},
  title = {{LLMEmbed: Rethinking Lightweight LLM's Genuine Function in Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.433},
}
Figure