Frame: Form-Restricted Authoring with Model-Based Evaluation

Nan Jiang, Mohammed Mohsin Hussain, Onkar Tiwari, Ankit Jain, Thong Q. Nguyen, Abhishek Agarwal, Vishal ChowdharyarXiv 2026 · arXiv

TL;DR

The authors introduce FRAME, a new benchmark for "form-restricted report generation"—the task of updating a report with new evidence while strictly adhering to a prior edition's structure. Their evaluation of five state-of-the-art LLMs reveals a striking "structural parity" where open-source models match proprietary ones in replicating layouts, yet lag significantly in information coverage and factual faithfulness.

What Problem Does This Paper Address?

In professional settings like financial analysis or academic research, writing is rarely a "blank page" exercise. Analysts often take a previous quarterly report and "plug in" new data, maintaining the same sections, tables, and flow.

Current LLM benchmarks focus on open-ended generation or standard summarization. These don't capture the specific challenge of form-restricted authoring: the model must simultaneously infer a complex document structure from a sample and ground new, often dense, evidence into that specific form. Existing metrics like ROUGE or BERTScore are also insufficient here, as they fail to account for hierarchical organization or subtle grounding errors in long-form documents.

Methods

The researchers developed a sophisticated pipeline to create the FRAME (Form-Restricted Authoring with Model-Based Evaluation) benchmark.

Benchmark Construction

They used a "temporal three-snapshot" approach. For a given company or research topic, they collected data at three points in time ($t_1, t_2, t_3$).

  1. Form-free ($t_1$): An LLM generates an initial report structure from scratch.
  2. Form-restricted ($t_2, t_3$): The LLM is forced to adapt the structure from the previous snapshot to new data. This ensures that successive reports are structurally aligned, making the earlier report a valid "form exemplar" for the next.
Figure 1: Overview of the FRAME benchmark construction pipeline.

Evaluation Suite

The paper moves beyond surface-level text matching with three human-validated metrics:

  • Structural Similarity: Uses a semantic tree-edit distance. Reports are converted into trees (sections, tables, etc.), and the algorithm calculates the cost to transform the generated tree into the "gold" tree. They found that using a Radial Basis Function (RBF) kernel to map embedding similarities better aligned with human intuition than simple linear mapping.
  • Coverage: Measured via Information Unit Recall (IUR). An LLM judge decomposes the gold report into atomic facts and checks how many are present in the generated output.
  • Faithfulness: A claim-level verification process where an LLM judge extracts propositions from the generated report and verifies them against the grounding documents.

Datasets

The FRAME benchmark consists of 120 instances across two primary domains:

  • Financial (EDGAR): 10-Q (Quarterly) and 10-K (Annual) filings. These are table-dense and follow stable conventions.
  • Academic (arXiv): Literature surveys on specific topics. These are prose-heavy and require multimodal understanding of figures and text.

The dataset is categorized into four levels of complexity: Financial Quarterly, Financial Annual, Academic Simple, and Academic Complex.

Table 1: Benchmark statistics per category. All counts are instance-level averages over the 30 instances in each category. Token counts use the o200k_base tokenizer.

Key Insights and Ideas

  • Structure is "Easy," Content is Hard: One of the most surprising findings is that open-source models (like Qwen3 and Phi-4) are nearly as good as GPT-5 at mimicking a report's layout. However, they fail significantly when it comes to actually finding the right data (coverage) and stating it accurately (faithfulness).
  • The "Long Context" Penalty: As grounding documents get longer and outputs grow, performance across all metrics tends to degrade. GPT-5 was the only model to maintain stable faithfulness as report length increased.
  • Tables are Anchors: Models perform better on financial reports than academic ones. The authors suggest that standardized, numerical, and table-heavy data provides clearer "hooks" for the model to ground its claims compared to flexible, prose-heavy academic papers.
  • RBF Mapping Matters: In developing their structural metric, the authors noted that text embeddings often cluster in a high-similarity range. Using an RBF projection (visualized below) helps amplify fine-grained differences, making the automated metric much more discriminative and human-aligned.
Figure 3: Studied projection functions mapping cosine similarity to calibrated semantic scores.

Key Results

The researchers evaluated five models: GPT-5, Claude-4.5-Sonnet, Qwen3-VL-30B, Phi-4-multimodal, and Kimi-VL-A3B.

  • Structural Parity: All models scored in a similar range for structural similarity (roughly 0.2–0.4).
  • The Proprietary Gap: GPT-5 and Claude-4.5 significantly outperformed open-source models in content. For example, in the "Overall" category, GPT-5 achieved a Coverage of 0.779 and Faithfulness of 0.914, while the best open-source model (Qwen3) trailed at 0.548 and 0.545, respectively.
  • Human Alignment: The automated metrics showed high correlation with human experts, with the coverage metric achieving a Quadratic Weighted Kappa (QWK) of 0.919.

Limitations and Future Work

The authors acknowledge two main limitations:

  1. Domain Scope: The benchmark is currently limited to finance and academia. Other structured domains like legal briefs or clinical reports might exhibit different patterns.
  2. LLM-as-Judge: While validated against humans, the evaluation still relies on a high-reasoning model (GPT-5), which may inherit specific model biases.

Future work could involve expanding the benchmark to more diverse professional domains and exploring whether specialized fine-tuning can close the massive coverage and faithfulness gap currently seen in open-source models.

Summary updated

Abstract

Many professional reports reuse the structure of prior editions while updating the content. We formalize this setting as form-restricted report generation: given a sample report and new grounding documents, generate a report that preserves the sample's form while accurately reflecting the new evidence. We introduce FRAME, a benchmark of 120 EDGAR- and arXiv-derived instances across Financial Quarterly, Financial Annual, Academic Simple, and Academic Complex categories, built with a temporal three-snapshot pipeline. We evaluate outputs with three human-validated metrics: structural similarity via semantic tree-edit distance, coverage via information unit recall, and faithfulness via claim-level grounding verification. Across five LLMs, open-source models nearly match proprietary models on structure but lag on coverage and faithfulness; all models perform better on financial than academic reports, and performance declines with longer outputs and more complex grounding.
BibTeX
@misc{jiang2026frame,
  author = {Nan Jiang and Mohammed Mohsin Hussain and Onkar Tiwari and Ankit Jain and Thong Q. Nguyen and Abhishek Agarwal and Vishal Chowdhary},
  title = {{Frame: Form-Restricted Authoring with Model-Based Evaluation}},
  year = {2026},
  eprint = {2611.08921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08921},
}
Figure