Scale Versus Skill: A Test of Why ESG Raters Disagree

Commercial ESG rating agencies (firms that score companies on environmental, social and governance performance) often give the same company very different grades. This disagreement already has measurable effects on markets. Explanations usually stop at "the agencies use different weights." This note proposes a more specific, testable mechanism. A firm can fall short of best practice in two separate ways: it can convert inputs into outputs poorly given its size (a skill problem), or it can be the wrong size for its technology (a scale problem). If some agencies' methods mainly reward one and some mainly reward the other, they should disagree most about firms whose two kinds of efficiency point in opposite directions. The direction of the disagreement, meaning which agency rates the firm higher, should flip when the firm's pattern flips.

The tool for separating skill from scale is Data Envelopment Analysis (DEA). Among comparable firms, DEA traces a best-practice frontier of outputs per input and scores each firm by its distance from that frontier. The standard decomposition splits a firm's overall shortfall into pure technical efficiency (skill at its current size) and scale efficiency (closeness to the best size). The prediction is specific to agency pairs, and it has a sign. That makes it falsifiable, and neither firm size nor a generic "noise" story can reproduce it. If it holds, it also opens a route to a known puzzle: rating disagreement raises bond spreads but attracts green equity investors.

Core Claim

Let $r_{iat}$ be agency $a$'s percentile rank of firm $i$ in year $t$. Let $p_{it}$ and $s_{it}$ be the firm's log pure technical efficiency and log scale efficiency, each standardized within industry-year. Each agency has a "scale tilt" $\tau_a$: how much more its ranks load on $s$ than on $p$. The tilt is estimated on held-out folds of firms so that it is not fit to the outcome. The claim is that the signed rank gap between two agencies follows

$$ r_{iat} - r_{ibt} = \alpha_{ab,jt} + \gamma\,(\tau_a - \tau_b)\,(s_{it} - p_{it}) + X_{it}'\beta_{ab} + \varepsilon_{iabt}, $$

with $\gamma > 0$. Here $\alpha_{ab,jt}$ is an agency-pair by industry-year fixed effect, which absorbs every fixed difference in how the two agencies weight things. $X_{it}$ holds size and accounting controls, and their coefficients $\beta_{ab}$ are allowed to differ by agency pair.

The interaction term is the new part. It says disagreement grows with two things together: how differently the two agencies weigh scale versus skill, and how far apart the firm's scale and skill efficiencies are. The sign of the gap reverses when $s_{it} - p_{it}$ changes sign.

Why This Gap Is Real

The two literatures each supply half of the argument.

On the efficiency side, several papers find that scale and skill come apart in practice. A study of zero-waste city clusters uses an undesirable-output super-efficiency model, a DEA variant that treats pollution as a bad output. It finds pure technical efficiency staying above 1 while scale efficiency stays below 1, so scale is the binding constraint (Measuring the Construction Efficiency of Zero-Waste City Clusters…). Chinese new-energy companies show the opposite pattern: scale efficiency reaches up to 0.9 while technical efficiency declines (Research on High Quality Evaluation… China Energy Finance). Mega-banks such as ICBC rank 15th of 20 on comprehensive performance, behind small city banks (Financial Performance Evaluation of Listed Commercial Banks in China). These papers motivate the proposal: they show the two components vary independently across real organizations. None of them involves external ratings.

On the rating side, disagreement across six Chinese-market agencies widens corporate bond spreads, more so for small firms and through the environmental and social pillars (Beyond the Rating). Separately, disagreement raises green institutional ownership by roughly 16.5% to 23.5% (Empowerment or detriment). Both papers treat firm size only as a linear control or a sample split. Neither asks whether the agencies are reading different aspects of how the firm operates.

The closest link between ratings and efficiency is a comparison of DJSI-listed banks with other banks using DEA scale and technical efficiency (Are Sustainable Banks Efficient and Productive?). It uses a single binary sustainability label, so disagreement cannot be studied. A single-agency study finds that total assets depress ROE and ROA while the Wind ESG score raises them (Impact of ESG Score on Firms' Performance…). It is also one cross-section with raw size.

The proposal therefore differs from all of these in one way. It asks whether the pattern of disagreement between specific agency pairs is predicted by the firm's position relative to the frontier, not by its size or its average rating.

Two further findings make the mechanism plausible:

  • Ratings are simple weighted sums. Refinitiv composites are close to exact linear rollups of category scores, with an adjusted $R^2 \approx 1.0$ (Investigate The ESG Score Methodology). This implies each agency carries a stable, estimable tilt.
  • Ratings track fundamentals. Fundamentals explain 53.9% of the variance in Refinitiv scores, and the environmental pillar is the most predictable (Heterogeneous Ensemble for ESG Ratings Prediction). This implies ratings respond to operational reality rather than to definitions alone.

Research Program

Building the efficiency measures. The first component is a firm-year efficiency panel for Chinese A-share firms from 2009 to 2023. It will be built from CSMAR financials, matching the universe used in the two disagreement papers.

  • Frontiers. These are estimated within industry-year. Inputs are total assets, employees and operating cost. Outputs are revenue and operating profit. Where Scope 1 and 2 emissions are disclosed, they enter as undesirable outputs in the undesirable super-efficiency model, which yields the pure technical and scale components.
  • Keeping it non-circular. The frontier deliberately excludes any rating-derived input, so the link between efficiency and ratings is not built in.
  • Avoiding uninformative scores. DEA loses its ability to distinguish firms when there are few firms per dimension; many units score exactly 1.0 (Carbon emissions management efficiency evaluation…). Two techniques are reused to address this. Thin industries get the regularized aggregation of Endogenous Aggregation of Multiple DEA Scores, which restored ranking where plain models scored every unit 1.0. The several emissions measures are merged into one indicator using the evidential-reasoning integration from the carbon paper.
  • Returns to scale. We also record whether each firm sits on the increasing or decreasing side of returns to scale. Below optimal size and above it are different scale problems, and the threshold evidence from sports enterprises shows effects can switch sign at a size cutoff (The Dynamic Impact of Diversification Strategy…).

Estimating agency tilts and testing the claim. For each of the six agencies, we regress percentile ranks on $p$ and $s$ within industry-year, using cross-fitting: tilts are estimated on half the firms and used on the other half. We then estimate the gap equation over all 15 agency pairs. Point-in-time ratings are used wherever the vendor provides them, because retroactive methodology recalculation introduces look-ahead bias (Investigate The ESG Score Methodology).

Reconciling the market puzzle (conditional stage). This stage runs only if the core claim survives. Each firm's disagreement is split into a part predicted by the scale-skill structure and a residual. The hypothesis is that sophisticated green investors respond to the structured part, because it is information about how the firm operates. Bond spreads, in contrast, respond mainly to the residual, which looks like noise to creditors. If correct, this explains why the bond-spread and green-ownership papers reach opposite conclusions. This stage is a hypothesis, not the main claim. The program is still worthwhile if this stage fails.

Baselines

Naive reference: pair-level weighting only. This model contains only the agency-pair by industry-year fixed effects. It is the formal version of "agencies just weight things differently." Every richer model must improve on its out-of-sample prediction of the signed gap, using the same held-out firm folds. No reported result was found on Lacuna for predicting signed pairwise ESG rating gaps, so this reference is computed fresh.

Size baseline. This model adds log assets, log assets squared, and size-decile indicators, each interacted with the agency pair. It lets each pair of agencies have its own size tilt. The firm-size heterogeneity documented in Beyond the Rating is exactly what this baseline captures. It gets the same folds and controls as the main model.

Strongest existing method: flexible fundamentals model. This is the strongest competitor and the greatest risk to the proposal. It is a heterogeneous ensemble of gradient-boosted trees and a feed-forward network in the style of Heterogeneous Ensemble for ESG Ratings Prediction. That paper reports $R^2$ of 53.9% for composite score levels, against about 27% for linear baselines and about 6% for industry means. For fairness, the ensemble is trained to predict the signed gap directly. It receives the raw DEA inputs and outputs, the size variables, and agency-pair identity, so it could in principle learn any scale pattern itself. It gets the same hyperparameter-search budget as our second-stage model and the same folds.

What counts as beating it. We add the single cross-fitted interaction term to the ensemble's out-of-fold residuals. The proposal beats this baseline if that term is significant and improves held-out $R^2$ on firm folds and on held-out years. A gain that appears only in-sample does not count.

Ablations. Two ablations isolate the key ingredient.

  • Placebo tilts. Agency tilts are permuted across agencies 1,000 times, giving a null distribution for $\gamma$. This tests whether the result depends on the agency-specific structure.
  • Ratio substitute. The DEA decomposition is replaced with simple ratios: revenue per asset as "skill" and log assets as "scale." This tests whether the frontier itself carries anything that crude ratios do not.

Why not just…?

Why not just use firm size? Size can give each agency pair its own slope. It cannot make the sign of a gap flip between two firms of identical size whose scale and skill efficiencies rank in opposite orders. The decisive test restricts the sample to firms within narrow size bins and checks whether $\gamma$ survives there. If it vanishes within bins, the size story wins and we will say so. The bank evidence, where the largest institution ranks 15th of 20 (Financial Performance Evaluation…), suggests size and frontier position diverge enough for the test to have power.

Why not just let a gradient-boosted model learn it from fundamentals? This is a real threat. Given pair identity and the same raw inputs, a flexible learner could rediscover an agency-specific scale pattern. The residual test above settles the question. If the interaction adds nothing beyond the ensemble's predictions, the honest conclusion is that frontier structure is learnable from fundamentals and DEA adds nothing for prediction. What would remain is an interpretable account: one estimated number per agency (its tilt), which a black-box model does not provide. We would report the weaker conclusion rather than claim predictive gains.

Why not just treat disagreement as agency definitional noise? The pair-by-industry-year fixed effects absorb all fixed definitional differences. The claim only concerns variation that depends on the firm's efficiency profile, within those cells. Two published facts argue against pure noise: the 53.9% predictability of scores from fundamentals, and green investors' stronger response for low-transparency firms (coefficient 0.203, Empowerment or detriment). Still, the placebo permutation is the direct test.

Why not just use intensity ratios instead of DEA? Ratios are cheaper and need no frontier. They also treat each input or output separately and cannot say whether a firm is too large for its technology. The ratio-substitute ablation compares the two directly. If ratios match DEA, we adopt ratios and drop the frontier machinery.

Decisive Checks

The program succeeds if four things hold:

  1. $\gamma$ is positive on held-out firm folds and on held-out years.
  2. $\gamma$ survives within size bins.
  3. $\gamma$ beats the placebo-tilt null distribution.
  4. $\gamma$ adds out-of-sample $R^2$ over the residuals of the flexible ensemble.

One sign check is sharp. Among firms with increasing versus decreasing returns to scale, the gap between a high-tilt agency and a low-tilt agency should reverse direction, and nothing in the size or noise accounts predicts that.

The outcome of the main test determines what follows:

  • If the main test passes, the market-reconciliation stage proceeds. It uses the bond panel for 2014 to 2023 and the green-investor panel for 2009 to 2022. The same controls and instrumental-variable designs as the original papers are used, so differences in coefficients can be attributed to the decomposition and not to specification.
  • If it fails, the program ends with a clean negative result: ESG raters do not differ in how they read scale versus skill.