TL;DR
fPLSA marries classical Probabilistic Latent Semantic Analysis (PLSA) with large language models (LLMs) to automatically discover latent, high-level semantic structures in document collections. By iteratively clustering segments and refining natural-language tag descriptions via an Expectation-Maximization (EM) loop, fPLSA learns expressive structural tags that significantly improve document reconstruction and boost problem-solving diversity in hierarchical sampling.
What Problem Does This Paper Address?
When humans solve complex problems or write stories, they don't just rely on raw words—they manipulate high-level concepts, solution blueprints, and structural schemas. Standard LLM prompting approaches (like Chain-of-Thought) rely heavily on human-engineered steps or one-shot prompting.
Existing automated approaches to discover these latent structures fall short:
- Traditional topic models (LDA, PLSA): Rely on bag-of-words lexical co-occurrences, missing sentence- and document-level semantics and failing on short text segments.
- LLM-based "generate-and-merge" methods: Ask an LLM in one shot to invent and consolidate topics, typically producing broad, overly generic themes (e.g., "Algebra and equations") rather than functional, reusable procedural steps.
The authors ask: Can an iterative, probabilistic framework guide foundation models to discover precise, reusable latent semantic structures across documents without supervision?
Methods
fPLSA reimagines classical PLSA using an LLM as both the segment classifier and the concept summarizer:
-
Generative Model Formulation: A document $d$ is broken into segments $x_k = w_{1...n}$. Rather than assuming words are conditionally independent given a topic, fPLSA models segment probabilities jointly: $$p_\Theta(w_{1...n}, x_k, d) = p(d) p(x_k|d) \sum_t p_\Theta(t | x_k, d) p_\Theta(w_{1...n} | t)$$ The model parameters $\Theta$ consist of a frozen LLM and a collection of natural-language descriptions $\theta_t$ for each latent tag $t$.
-
Iterative EM Optimization:
- E-step (Tag Assignment): Given current text descriptions $\{\theta_t\}_{t=1}^T$, the LLM assigns each segment $x_k$ to a tag, conditioned on surrounding context segments within a window $W$.
- M-step (Description Refinement): The algorithm aggregates segments assigned to each tag, samples representative examples, and prompts the LLM to update $\theta_t$ with a concise summary describing what those segments share in common.
- Hierarchical Sampling: To solve new problems, fPLSA fits a simple bigram transition model $p(t_k | t_{k-1})$ over tag sequences from training data. It samples an "outline" sequence of tags and prompts the LLM to generate the final response conditioned on this structural outline.
┌──────────────────────────────┐
│ Segmented Document Collection│
└──────────────┬───────────────┘
│
┌──────────▼──────────┐
┌──────►│ E-Step │ (LLM assigns segments to tags
│ │ Tag Assignment │ using surrounding context)
│ └──────────┬──────────┘
│ │
M-Step ▼
(LLM updates ┌──────────────────┐
descriptions) │ M-Step │
│ │ Tag Optimization │
└──────────┴──────────────────┘
│ Converged Tags & Transitions
▼
┌──────────────────────────────┐
│ Hierarchical Sampling │
│ (Outline -> Final Generation)│
└──────────────────────────────┘
Datasets
The authors evaluated fPLSA across three benchmarks:
- WritingPrompts: 100 randomly sampled human-written creative stories (segmented into scenes; 100 tags).
- MATH: 1,000 sampled high school competition problems and step-by-step solutions across 7 subjects (Prealgebra to Precalculus; 100 tags).
- Big-Bench Hard (BBH): Step-by-step solutions for 12 multi-step reasoning tasks generated by automatic Chain-of-Thought inference (50 tags).
No new benchmarks were created; existing text/solution corpora were repurposed for unsupervised structural discovery.
Key Insights and Ideas
- Iterative Refinement Prevents Topic Collapse: Direct LLM prompting often bunches heterogeneous segments into generic buckets. Iterative EM allows the LLM to separate broad topics into actionable, fine-grained procedural strategies (e.g., distinguishing between "Perform algebraic manipulations" and "Utilizes specific theorems like De Moivre's").
- Context Matters for Segment Semantics: By conditioning tag assignment on adjacent context segments rather than isolated text spans, tags capture narrative and logical transitions instead of isolated vocabulary.
- Cross-Task Tag Transfer Promotes Diverse Thinking: When sampling solutions for a specific task using tags learned from a different task, performance frequently matched or exceeded native tags. Out-of-domain tags force the LLM to explore unconventional solution paths ("think out of the box"), increasing the search coverage.
Key Results
- Text Reconstruction Likelihood: Conditioning on fPLSA tags yielded substantially higher reconstruction log-likelihood than unconditioned generation (+0.6 to +1.4) and outperformed traditional LDA, LDA+LLM labeling, and one-shot LLM Prompting baselines by 0.2 to 0.4 on WritingPrompts and MATH.
- Problem Solving (Hits@50 Accuracy): Hierarchical sampling using fPLSA outlines significantly improved multi-solution diversity and coverage:
- MATH: Improved average Hits@50 from 62.9% (direct sampling) and 65.6% (LLM outline generation) to 72.3% (+9.4% over direct sampling).
- BBH: Improved average Hits@50 from 74.5% (direct) and 82.3% (LLM outline) to 91.1% (+16.6% over direct sampling).
- Outperformed both traditional LDA (+2 to +10 points) and prompting-based taggers (+1 to +11 points) across most tasks.
Limitations and Future Work
- Transition Modeling: The outline generator currently relies on a simple discrete bigram model ($p(t_k | t_{k-1})$), which misses longer-range dependencies across complex multi-step reasoning plans.
- Fixed Hyperparameters: The number of tags $T$ and the iteration cap (30 iterations) are fixed heuristics rather than dynamically determined.
- Cost of EM Iterations: Querying proprietary LLMs (e.g., GPT-4) repeatedly over all segments across 30 EM iterations introduces significant compute and API latency overhead.
- Future Directions: Incorporating verifier or reward models during hierarchical search, and generating automatic task guidelines for unfamiliar domains from a handful of unannotated demonstrations.
Summary updated