fLSA: Learning Semantic Structures in Document Collections Using Foundation Models

Weijia Xu, Nebojsa Jojic, Nicolas Le Roux

TL;DR

This paper introduces fLSA, a novel method that uses Large Language Models (LLMs) to learn latent semantic structures within document collections by iteratively clustering and tagging document segments. This approach significantly improves text reconstruction and aids LLMs in generating more accurate solutions for complex tasks through hierarchical sampling, outperforming existing methods.

What Problem Does This Paper Address?

Current approaches to leveraging LLMs for complex tasks often rely on human-crafted few-shot examples and chain-of-thought prompting. While effective, this requires manual effort to identify and demonstrate problem-solving strategies. Humans, however, naturally learn by analyzing examples, inducing underlying strategies (latent semantic structures), and applying them to new problems. This paper aims to bridge this gap by developing an automated method, fLSA, that can discover these latent semantic structures within document collections, making them accessible for LLMs to use without explicit human guidance. This is particularly relevant for tasks requiring multi-step reasoning or planning, where understanding the underlying structure is crucial.

Methods

fLSA is a foundation-model-based iterative algorithm that combines principles from traditional Latent Semantic Analysis (LSA) and modern LLMs. It operates similarly to an Expectation-Maximization (EM) algorithm.

  • Iterative Clustering and Tagging: The core idea is to iteratively refine "tags" that represent semantic concepts within document segments.
  • LLM Integration: Instead of traditional statistical methods, fLSA leverages LLMs (specifically GPT-4 and Qwen-2.5-7B) with frozen parameters. The LLM is used for two key tasks:
  • E-step (Tag Assignment): Given existing tag descriptions, the LLM assigns a tag to each document segment. This assignment considers not only the current segment but also its neighboring segments (context window) to capture document-level context.
  • M-step (Tag Description Update): After segments are assigned tags, the LLM is prompted to generate new, more informative descriptions for each tag by summarizing the aggregated segments. This process refines the semantic meaning of each tag.
  • Generative Model: fLSA models the probability of a sequence of words within a segment conditioned on both the segment and the document it belongs to, and then on a specific tag. This is a departure from traditional LSA, which models word probabilities independently.

Datasets

The paper evaluates fLSA on three diverse datasets:

  • WritingPrompts: A dataset of 300K human-written stories paired with prompts, used for story writing tasks.
  • MATH: A benchmark containing math problems and their solutions across seven subjects (Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus).
  • Big-Bench Hard (BBH): A benchmark comprising 23 challenging multi-step reasoning tasks. For this study, they used 12 tasks from a previous work and their corresponding problem solutions.

Key Insights and Ideas

  • LLMs as Structure Discoverers: The paper demonstrates that LLMs can be effectively used not just for task execution but also for uncovering underlying semantic structures in data, mimicking human learning processes.
  • Contextual Tagging is Key: Incorporating document-level context (neighboring segments) during tag assignment significantly improves the quality and informativeness of the learned tags.
  • Hierarchical Sampling Power: Learned tags can serve as a powerful outline for hierarchical sampling, guiding LLMs to explore more relevant solution paths and significantly increasing the chances of finding correct answers for complex reasoning tasks.
  • Beyond Traditional Topic Models: fLSA's tags are shown to be more informative and detailed than those generated by traditional topic models like LDA, even when augmented with LLMs, because they leverage richer contextual information.

Key Results

  • Reconstruction Likelihood: fLSA tags improve reconstruction log-likelihood by 0.7–1.4 points over a "No Tag" baseline and consistently outperform traditional LDA and LDA+LLM baselines across all three datasets.
  • Hierarchical Sampling Performance:
    • On MATH and BBH, hierarchical sampling with fLSA tags leads to significantly higher Hits@K accuracy compared to direct sampling without tags (+10.0 points on MATH, +16.6 points on BBH on average).
  • fLSA also outperforms other tagging approaches like TradLDA and Prompting in hierarchical sampling, achieving 2-11 points higher average Hits@K accuracy.
  • Smaller LLM Efficacy: fLSA can be effectively used with smaller LLMs (like Qwen-2.5-7B), demonstrating its adaptability, although GPT-4-learned tags yield slightly better results.

Limitations and Future Work

  • Tag Similarity: Some learned tags might be semantically similar and could potentially be merged. The authors suggest a "tag fusion" step in the EM algorithm as future work.
  • LLM Agnosticism vs. Testing: While fLSA is designed to be LLM-agnostic, it was primarily tested on GPT-4. Future work could explore its performance with a wider range of LLMs, especially smaller ones.
  • Bias in LLMs: A significant risk is that fLSA tags might reflect undesirable biases present in the LLM. Integrating bias detection and mitigation techniques is suggested for future research.
  • Number of Tags: Ablation studies suggest that learning a sufficient number of tags (around 50-100) is beneficial for effective expansion of the output space, with minimal gains beyond 100 tags.

Summary updated

Abstract

Humans have the ability to learn new tasks by inferring high-level concepts from existing solution, then manipulating these concepts in lieu of the raw data. Can we automate this process by deriving latent semantic structures in a document collection using foundation models? We introduce fPLSA, a foundation-model-based Probabilistic Latent Semantic Analysis (PLSA) method that iteratively clusters and tags document segments based on document-level contexts. These tags can be used to model the structure of given documents and for hierarchical sampling of new texts. Our experiments on story writing, math, and multi-step reasoning datasets demonstrate that fPLSA tags help reconstruct the original texts better than existing tagging methods. Moreover, when used for hierarchical sampling, fPLSA produces more diverse outputs with a higher likelihood of hitting the correct answer than direct sampling and hierarchical sampling with existing tagging methods.
BibTeX
@inproceedings{xu2025flsa,
  author = {Weijia Xu and Nebojsa Jojic and Nicolas Le Roux},
  title = {{fLSA: Learning Semantic Structures in Document Collections Using Foundation Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure