A widely used way to say how "sophisticated" a traded product is, called PRODY, takes the average income per person of the countries that export it. Goods exported mostly by rich countries score high, and a country's export basket is then judged by how many high-scoring goods it contains. The measure is circular by construction. It uses income to rate products, and it is then used to explain income. So a high score cannot tell us whether a product needs rare technical capability or just happens to be shipped by wealthy, resource-rich economies. Every competing measure is also computed from the same export records, so there is no outside yardstick for deciding which measure is right.
We propose to build that yardstick using the Delphi method, a structured way of polling experts. Experts rate items anonymously, see a summary of the group's answers, and revise over several rounds until their ratings stabilize. Here, engineers and industry specialists would rate what each product takes to make, without seeing who exports it or at what income level. A large language model (LLM) persona panel, calibrated against the human panel, extends the ratings to the full product list. The result is a product sophistication score that is independent of income by design. It lets us measure, product by product, where PRODY reflects know-how and where it reflects wealth. It also lets us run the long-standing claim that "sophisticated exports predict growth" without the circularity.
Core Claim
Our claim is that income-based sophistication mixes two different things. One is the capability required to make a good. The other is the endowment (natural resources or climate) that lets rich countries export it. An income-blind expert rating can separate the two. Concretely, we expect three results. First, PRODY and expert-rated capability will disagree most sharply for products that depend on natural endowments. Second, a trade-network measure that uses no income data, the Fitness-Complexity method (FCM), will agree with experts better than PRODY does. Third, a country index built from expert scores will predict later income growth at least as well as the PRODY-based index, despite containing no income information.
If all three hold, the expert score becomes a validation target that economic complexity research currently lacks. If the third fails, that is also informative. It would suggest that PRODY's predictive power comes partly from its built-in income signal.
Relation to Prior Work
The economic complexity work in the file motivates the gap. PRODY-based technical sophistication indices are used as outcome variables, for instance to show that geographical-indication certification raises agricultural export sophistication, with a benchmark coefficient of 0.140 that falls to 0.121 at the 85th percentile (Does Geographical Indication Certification Increase the Technical Complexity of Export Agricultural Products?). That study treats the index as ground truth. We ask whether the index deserves that trust, especially for agricultural goods, where endowments matter most.
The FCM avoids income by iterating between country fitness and product complexity on the network of which countries export which products. It is used to study the middle-income trap (The Middle-Income Trap and the Coping Strategies From Network-Based Perspectives), export recommendation (Economic Complexity Based Recommendation Enhance the Efficiency of the Belt and Road Initiative), and mutual influence between countries (Economic Trading Susceptibility: Constructing Networks of Mutual Influence through the Fitness of Countries). But it is still validated only against trade and income data.
The closest work decomposes the trade matrix into latent country capabilities and product requirements with a sparse Bayesian factorization (S3R-IBP), then names the factors by inspecting top products (Economic complexity unfolded: Interpretable model for the productive structure of economies). The obvious next step would be to have a Delphi panel name those factors. We deliberately do something different. We do not ask experts to interpret factors extracted from trade data, because that inherits the trade data's biases. Instead we elicit capability judgments independently of trade and income, then use them to test PRODY, FCM, and the factorization alike.
The Delphi literature supplies the method. The Human-AI Hybrid Delphi pairs an LLM evidence synthesis with about six senior experts. It uses a four-tier consensus scale that keeps conditional "it depends" judgments, and it reports 95% replication of historical consensus items (The Human-AI Hybrid Delphi Model). We borrow its panel size and consensus tiers. Evidence-grounded persona panels with bounded-confidence opinion revision reached Pearson $r=0.922$ against human raters, compared with $r=0.716$ for a single LLM judge, on generative-interface evaluation (Beyond a Single Judge). We reuse this architecture to scale human ratings to the full product list.
The file also contains a constraint that shapes our design. A purely LLM-run Delphi showed a positivity bias and training-cutoff effects (An LLM-based Delphi Study to Predict GenAI Evolution). LLMs also likely "know" which countries export which goods, which could quietly reintroduce the income signal we are trying to exclude. We treat that leakage as the main threat and test for it explicitly.
Research Program
Rating rubric. A small human pilot builds and validates the rubric, screening items with the content validity ratio. This is the share of panelists calling an item essential, rescaled to run from −1 to 1, as used for the VISION-Cog battery (Vu et al.). The candidate rubric scores each product on five dimensions: how much tacit know-how production requires, how many specialized inputs and suppliers it depends on, the precision and quality-control demands, the certification and regulatory burden, and dependence on natural endowment. Endowment dependence is rated separately and deliberately kept out of the capability score. It is the variable we expect to explain PRODY's errors.
Blinding. Products are presented as short process descriptions, meaning inputs, transformations, and equipment. Country names, trade statistics, and brand associations are removed.
Human Delphi panels. Four sector panels (agriculture and food, materials and chemicals, machinery and electronics, light manufacturing) each have six to eight industrial engineers, production managers, and sector economists. Each panel rates about 40 products. The roughly 160 products are sampled evenly across quintiles of PRODY–FCM rank disagreement, so the sample over-represents exactly the cases where the two measures conflict. Panels run up to three anonymous rounds with median and interquartile feedback. Final judgments are recorded on the four-tier consensus scale, and conditional rationales are coded.
LLM persona panel. A panel of personas grounded in sector evidence, following the persona-panel design, rates all 770 SITC products in the historical panel of Zhou et al. using the same blinded descriptions. It is calibrated only against human ratings on a training split of the 160 human-rated products.
Country index. The country-level analogue of PRODY's export index is
$$S_c = \sum_p \frac{x_{cp}}{X_c} E_p ,$$
where $x_{cp}$ is country $c$'s exports of product $p$, $X_c$ is its total exports, and $E_p$ is the expert capability score. This index is identical in form to the PRODY-based index except that $E_p$ contains no income information.
Baselines
PRODY. The naive reference point is PRODY itself, computed from Observatory of Economic Complexity SITC trade flows and external GDP-per-capita series, exactly as defined in the GI study. It is the measure under test, and it is the default in applied work.
FCM complexity. The strongest existing income-free competitor is FCM product complexity, computed on the same data with the same export-specialization threshold (a country counts as an exporter when its revealed comparative advantage is at least 1). It removes PRODY's circularity using network structure alone. If FCM agrees with human experts as well as the persona panel does, the expert pipeline adds little for rating products, though it would still validate FCM.
Latent-requirement score. We compute a score from the S3R-IBP factorization as the number of latent capabilities each product requires. For its native task, predicting which new products countries will export within four years, the file reports 32–44% accuracy against a 7% random baseline (Economic complexity unfolded). Adding expert scores as product features counts as beating it only if accuracy rises above that range on the same split.
Single LLM judge. The strongest cheap competitor to the persona panel is one frontier-model call per product with the same rubric and descriptions, plus a five-call average. It gets the same tuning budget, meaning the same number of prompt iterations on the calibration split. For reference, on UI evaluation the file reports $r=0.716$ for the single judge against $r=0.922$ for the persona panel (Beyond a Single Judge). No reported result was found on Lacuna for either method on product sophistication.
Ablations. Three ablations isolate the key ingredients. The first un-blinds descriptions by adding product names and example exporters, which isolates the effect of blinding. The second folds endowment dependence into the capability score, which isolates the separate endowment axis. The third drops the revision rounds, which isolates deliberation.
On product rating, beating a baseline means higher Spearman correlation with held-out human consensus ratings, with bootstrap confidence intervals that do not overlap.
Why not just…?
Why not just use FCM, which already removes income? FCM removes income as an input, but it is judged only against trade and income outcomes, so nobody can say whether it measures capability. The experiment settles this directly. If FCM matches the human panel as well as the persona panel does, we report FCM as validated and treat the LLM panel as unnecessary. This is a real possible outcome, not a strawman.
Why not just use existing R&D-intensity or technology-class groupings of industries? These groupings are a genuine threat for coarse sorting. However, they are defined for whole industries rather than individual products, and they are built largely from rich-country R&D data. We include such a grouping, mapped to SITC codes, as an extra comparator. It is displaced only if the expert scores separate products within the same technology class, particularly within agriculture and food, where the GI results sit.
Why not just ask one strong LLM? Evidence from UI evaluation favors panels, but that domain is different. The stronger worry is that any LLM reproduces PRODY from memorized trade patterns. We test this by having the model predict the average income of each product's exporters from the blinded description. If that prediction explains most of the variance in the LLM's capability ratings, we report the LLM panel as contaminated and restrict conclusions to the 160 human-rated products.
Why not just test whether PRODY predicts growth? It does so partly by construction, which is the problem. Only an income-free score can distinguish know-how from income echo.
Proposed Experiments and Decisive Checks
Endowment test. On human-rated products, we regress the PRODY rank minus the expert-capability rank on rated endowment dependence. We predict a positive, significant slope, largest for agricultural and mineral goods. This directly checks whether the GI study's high-end ceiling reflects PRODY mismeasuring endowment-driven goods rather than a failure of international recognition.
Agreement test. We compare Spearman correlations of PRODY, FCM, the latent-requirement score, the single judge, and the persona panel with held-out human consensus. We predict that FCM will beat PRODY. The panel must beat the single judge and survive the leakage probe to be used at scale.
Growth test. On the 1962–2000 panel of Zhou et al. and the longer SITC series, we regress ten-year per-capita growth on initial $S_c$, the PRODY-based index, and FCM fitness, each alone and jointly, controlling for initial income and resource-rent share. No reported result on this comparison was found on Lacuna. The decisive check is whether $S_c$ keeps its predictive weight once the PRODY-based index is included. If it does, capability is doing the work. If only the PRODY-based index survives, its predictive power is partly an income echo.
Stability check. We rerun the human Delphi with a second, independent set of panels on a 40-product overlap. If inter-panel rank agreement is low, expert capability is not a stable construct, and the project reports that negative result.
References
- Does Geographical Indication Certification Increase the Technical Complexity of Export Agricultural Products?
- The Middle-Income Trap and the Coping Strategies From Network-Based Perspectives
- Economic Complexity Based Recommendation Enhance the Efficiency of the Belt and Road Initiative
- Economic Trading Susceptibility: Constructing Networks of Mutual Influence through the Fitness of Countries
- Economic complexity unfolded: Interpretable model for the productive structure of economies
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- An LLM-based Delphi Study to Predict GenAI Evolution
- Feasibility, comprehensibility and acceptability of the VISION-Cog, a novel tool to assess cognitive impairment in visually impaired older adults: a cross-sectional pilot study in Singapore