TL;DR
The authors introduce FRAME, a new benchmark for "form-restricted report generation"—the task of updating a report with new evidence while strictly adhering to a prior edition's structure. Their evaluation of five state-of-the-art LLMs reveals a striking "structural parity" where open-source models match proprietary ones in replicating layouts, yet lag significantly in information coverage and factual faithfulness.
What Problem Does This Paper Address?
In professional settings like financial analysis or academic research, writing is rarely a "blank page" exercise. Analysts often take a previous quarterly report and "plug in" new data, maintaining the same sections, tables, and flow.
Current LLM benchmarks focus on open-ended generation or standard summarization. These don't capture the specific challenge of form-restricted authoring: the model must simultaneously infer a complex document structure from a sample and ground new, often dense, evidence into that specific form. Existing metrics like ROUGE or BERTScore are also insufficient here, as they fail to account for hierarchical organization or subtle grounding errors in long-form documents.
Methods
The researchers developed a sophisticated pipeline to create the FRAME (Form-Restricted Authoring with Model-Based Evaluation) benchmark.
Benchmark Construction
They used a "temporal three-snapshot" approach. For a given company or research topic, they collected data at three points in time ($t_1, t_2, t_3$).
- Form-free ($t_1$): An LLM generates an initial report structure from scratch.
- Form-restricted ($t_2, t_3$): The LLM is forced to adapt the structure from the previous snapshot to new data. This ensures that successive reports are structurally aligned, making the earlier report a valid "form exemplar" for the next.
Evaluation Suite
The paper moves beyond surface-level text matching with three human-validated metrics:
- Structural Similarity: Uses a semantic tree-edit distance. Reports are converted into trees (sections, tables, etc.), and the algorithm calculates the cost to transform the generated tree into the "gold" tree. They found that using a Radial Basis Function (RBF) kernel to map embedding similarities better aligned with human intuition than simple linear mapping.
- Coverage: Measured via Information Unit Recall (IUR). An LLM judge decomposes the gold report into atomic facts and checks how many are present in the generated output.
- Faithfulness: A claim-level verification process where an LLM judge extracts propositions from the generated report and verifies them against the grounding documents.
Datasets
The FRAME benchmark consists of 120 instances across two primary domains:
- Financial (EDGAR): 10-Q (Quarterly) and 10-K (Annual) filings. These are table-dense and follow stable conventions.
- Academic (arXiv): Literature surveys on specific topics. These are prose-heavy and require multimodal understanding of figures and text.
The dataset is categorized into four levels of complexity: Financial Quarterly, Financial Annual, Academic Simple, and Academic Complex.
Key Insights and Ideas
- Structure is "Easy," Content is Hard: One of the most surprising findings is that open-source models (like Qwen3 and Phi-4) are nearly as good as GPT-5 at mimicking a report's layout. However, they fail significantly when it comes to actually finding the right data (coverage) and stating it accurately (faithfulness).
- The "Long Context" Penalty: As grounding documents get longer and outputs grow, performance across all metrics tends to degrade. GPT-5 was the only model to maintain stable faithfulness as report length increased.
- Tables are Anchors: Models perform better on financial reports than academic ones. The authors suggest that standardized, numerical, and table-heavy data provides clearer "hooks" for the model to ground its claims compared to flexible, prose-heavy academic papers.
- RBF Mapping Matters: In developing their structural metric, the authors noted that text embeddings often cluster in a high-similarity range. Using an RBF projection (visualized below) helps amplify fine-grained differences, making the automated metric much more discriminative and human-aligned.
Key Results
The researchers evaluated five models: GPT-5, Claude-4.5-Sonnet, Qwen3-VL-30B, Phi-4-multimodal, and Kimi-VL-A3B.
- Structural Parity: All models scored in a similar range for structural similarity (roughly 0.2–0.4).
- The Proprietary Gap: GPT-5 and Claude-4.5 significantly outperformed open-source models in content. For example, in the "Overall" category, GPT-5 achieved a Coverage of 0.779 and Faithfulness of 0.914, while the best open-source model (Qwen3) trailed at 0.548 and 0.545, respectively.
- Human Alignment: The automated metrics showed high correlation with human experts, with the coverage metric achieving a Quadratic Weighted Kappa (QWK) of 0.919.
Limitations and Future Work
The authors acknowledge two main limitations:
- Domain Scope: The benchmark is currently limited to finance and academia. Other structured domains like legal briefs or clinical reports might exhibit different patterns.
- LLM-as-Judge: While validated against humans, the evaluation still relies on a high-reasoning model (GPT-5), which may inherit specific model biases.
Future work could involve expanding the benchmark to more diverse professional domains and exploring whether specialized fine-tuning can close the massive coverage and faithfulness gap currently seen in open-source models.
Summary updated