Introduction
When humans learn to solve complex problems, we do not simply memorize solutions; we induce high-level strategies. We recognize that a mathematical proof typically begins with variable initialization, proceeds through algebraic manipulation, and concludes with simplification. While modern Large Language Models (LLMs) excel at following specific instructions via techniques like Chain-of-Thought, they struggle to autonomously discover these underlying semantic structures in an unsupervised manner.
Traditional topic modeling techniques, such as Latent Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA), treat documents as "bags of words," effectively ignoring the sequence of ideas and the deep semantic context. This paper introduces fLSA (foundation-model-based LSA), a novel framework designed to bridge this gap. fLSA enables models to learn the latent semantic structure of a domain without human intervention, creating a high-level roadmap that enhances reasoning and generation.
Methodology: The fLSA Framework
The authors propose an iterative algorithm based on the Expectation-Maximization (EM) framework, specifically adapted for the capabilities of foundation models. The process consists of two primary stages:
- The E-Step (Contextual Tagging): Given a set of document segments and a list of current tag descriptions, the LLM assigns the most appropriate tag to each segment. Unlike traditional methods, fLSA considers both the segment itself and the surrounding "document-level context" (neighboring segments). This allows the model to understand a segment's specific role in the overall flow of a document.
- The M-Step (Semantic Summarization): The LLM reviews all segments assigned to a particular tag and generates a refined textual description. This description captures the shared characteristics and semantic essence of those segments, ensuring the tags evolve to be more representative of the data.
By repeating these steps, the model converges on a set of highly descriptive, semantically meaningful tags. Once these tags are established, the authors construct a dynamic transition model (a bigram model) to predict the likelihood of one tag following another. To solve new problems, the system employs hierarchical sampling: it first samples a sequence of tags to create a structural "outline" and then prompts the LLM to generate the final text based on that outline.
Experimental Setup
The researchers evaluated the fLSA framework across three distinct and challenging domains:
- WritingPrompts: A dataset comprising 300,000 human-written stories, used to test narrative structure discovery.
- MATH: A collection of high-school competition-level mathematics problems spanning seven subjects, including Algebra and Geometry.
- Big-Bench Hard (BBH): A benchmark consisting of 23 difficult reasoning tasks that require multi-step logic.
Key Insights and Results
Emergent Semantic Structure
The model successfully discovered meaningful stages in complex tasks without manual labeling. For example, in the MATH dataset, it distinguished between "Calculations/Simplifications" (Tag 24) and the "Presentation of final answer" (Tag 51). This contextual awareness allows fLSA to differentiate between identical mathematical notations based on whether they appear during initial setup or final conclusion—a feat impossible for bag-of-words models.
Performance Gains
The effectiveness of fLSA was measured through reconstruction likelihood and problem-solving accuracy:
- Reconstruction Likelihood: fLSA tags proved significantly more informative than traditional LDA or simple zero-shot LLM prompting, allowing for much higher accuracy in reconstructing original text spans.
- Problem Solving (Hits@K):
- On the MATH dataset, hierarchical sampling using fLSA tags improved accuracy from 62.9% (direct sampling) to 72.9%.
- On BBH, the improvement was even more pronounced, with accuracy jumping from 74.5% to 91.1%.
- fLSA consistently outperformed other tagging methods, such as TradLDA, by margins of 2 to 11 percentage points.
Discussion and Future Work
While fLSA represents a significant step forward in unsupervised structure learning, several areas for improvement remain. Currently, the number of tags must be set manually, and some discovered tags exhibit semantic redundancy. The authors suggest a "tag fusion" step in the EM algorithm to address this. Additionally, while smaller models like Qwen-2.5-7B perform well, the initial tag discovery process using massive models like GPT-4 remains computationally intensive.
In conclusion, fLSA demonstrates that by leveraging the semantic depth of foundation models within an EM framework, we can move beyond simple word-frequency models toward a more sophisticated understanding of document structure, significantly boosting the reasoning capabilities of AI systems.
Summary updated