FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents

Zheng Dou, Deqing Wang, Fuzhen Zhuang, Jian Ren, Yanlin HuarXiv 2025 · arXiv · OpenReview

Abstract

Scientific document representation learning provides powerful embeddings for various tasks, while current methods face challenges across three approaches. 1) Contrastive training with citation-structural signals underutilizes citation information and still generates single-vector representations. 2) Fine-grained representation learning, which generates multiple vectors at the sentence or aspect level, requires costly integration and lacks domain generalization. 3) Task-aware learning depends on manually predefined task categorization, overlooking nuanced task distinctions and requiring extra training data for task-specific modules. To address these problems, we propose a new method that unifies the three approaches for better representations, namely FLeW. Specifically, we introduce a novel triplet sampling method that leverages citation intent and frequency to enhance citation-structural signals for training. Citation intents (background, method, result), aligned with the general structure of scientific writing, facilitate a domain-generalized facet partition for fine-grained representation learning. Then, we adopt a simple weight search to adaptively integrate three facet-level embeddings into a task-specific document embedding without task-aware fine-tuning. Experiments show the applicability and robustness of FLeW across multiple scientific tasks and fields, compared to prior models.

1. Introduction

In the ecosystem of AI-driven research, document embeddings—numerical vectors representing the semantic essence of a paper—are foundational for search engines, recommendation systems, and automated categorization. However, existing methodologies encounter three significant limitations. First, most models produce a monolithic vector for an entire document, failing to distinguish between functionally distinct sections like a novel methodology versus preliminary background. Second, fine-grained models often rely on sentence-level labels that lack cross-domain robustness. Finally, current "task-aware" models require rigid manual categorization and additional training data to adapt to diverse query types.

FLeW (Facet-Level and Adaptive Weighted representation learning) addresses these issues by decomposing papers into three universal functional facets: Background, Method, and Result.

2. Methodology

The FLeW framework unifies citation structure and textual content through a four-stage pipeline: structural sampling, textual splitting, facet-specific pre-training, and adaptive weighted inference.

The FLeW framework: (a) Structural Sampling, (b) Textual Splitting, (c) Pre-training, and (d) Inferring.

2.1 Structural Sampling and Textual Splitting

FLeW constructs three distinct citation subgraphs based on citation intent (the functional reason for a citation) and frequency. To align textual content with these signals, the system utilizes a "Splitter" model—a fine-tuned Llama-3.1-8B—to segment standard abstracts into Background, Method, and Result facets.

2.2 Pre-training and Inference

Three separate encoders, initialized from SciBERT, are trained via contrastive learning on their respective facets. During inference, FLeW generates three vectors for a single document. Instead of simple averaging, it employs a grid search on validation data to determine optimal weights for specific downstream tasks, allowing for task-specific adaptation without further fine-tuning.

3. Experimental Results

FLeW was evaluated on the SciRepEval benchmark, which includes 19 tasks across four formats: Proximity (PRX), Regression (RGN), Query (QRY), and Classification (CLF).

Performance comparison of FLeW against state-of-the-art models on the SciRepEval benchmark across Proximity, Regression, Query, and Classification tasks.

As shown in the results, FLeW achieved the highest performance in 13 out of 19 tasks, with an average score of 60.81, surpassing established models such as SPECTER-2 (60.21) and SciNCL (59.52). The model demonstrated particular strength in proximity and regression tasks, where the nuanced weighting of facets proved critical.

4. Discussion

4.1 Mitigation of Position Bias

Standard encoders often suffer from "position bias," disproportionately focusing on the beginning of an abstract (typically the Background). By explicitly splitting the text and training facet-specific encoders, FLeW ensures that the "Result" facet is represented with equal fidelity, leading to significant gains in outcome-based retrieval.

4.2 Domain Generalization

While sentence-level labels vary significantly across disciplines, the "Background-Method-Result" triad is nearly universal in scientific communication. This allows FLeW to generalize across diverse fields. Evaluation on the MDCR dataset confirmed FLeW's dominance in STEM fields (Biology, Chemistry, Medicine), though it showed slightly less dominance in humanities where writing structures are less standardized.

5. Conclusion and Future Work

FLeW demonstrates that facet-level decomposition and adaptive weighting provide a robust, domain-generalized approach to scientific document representation. While the current implementation requires higher inference overhead due to the triple-encoder architecture, future work will explore learnable "routers" to automatically determine facet weights based on query strings, further streamlining the adaptive process.

Summary updated

Abstract

Scientific document representation learning provides powerful embeddings for various tasks, while current methods face challenges across three approaches. 1) Contrastive training with citation-structural signals underutilizes citation information and still generates single-vector representations. 2) Fine-grained representation learning, which generates multiple vectors at the sentence or aspect level, requires costly integration and lacks domain generalization. 3) Task-aware learning depends on manually predefined task categorization, overlooking nuanced task distinctions and requiring extra training data for task-specific modules. To address these problems, we propose a new method that unifies the three approaches for better representations, namely FLeW. Specifically, we introduce a novel triplet sampling method that leverages citation intent and frequency to enhance citation-structural signals for training. Citation intents (background, method, result), aligned with the general structure of scientific writing, facilitate a domain-generalized facet partition for fine-grained representation learning. Then, we adopt a simple weight search to adaptively integrate three facet-level embeddings into a task-specific document embedding without task-aware fine-tuning. Experiments show the applicability and robustness of FLeW across multiple scientific tasks and fields, compared to prior models.
BibTeX
@misc{dou2025flew,
  author = {Zheng Dou and Deqing Wang and Fuzhen Zhuang and Jian Ren and Yanlin Hu},
  title = {{FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents}},
  year = {2025},
  eprint = {2509.07531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.07531},
}
Figure