When evaluating complex generative models—whether they produce images, text, video, or biological structures—we rarely have a straightforward mathematical loss function that perfectly captures "quality," "fairness," or "realism." To bridge this gap, the machine learning community heavily relies on proxy models. These are pre-trained classifiers, vision-language models (VLMs), or natural language inference (NLI) engines tasked with acting as automated judges. For instance, we might use an Inception network to score image realism, a CLIP model to evaluate text-to-image alignment, or a structural algorithm to verify a generated protein.
However, this approach introduces a fundamental vulnerability: the evaluation metric ceases to be an objective ground truth. Instead, it becomes a reflection of the proxy model's own architectural quirks, training data skews, and systemic biases. As Hirono Okamoto et al. observe in Dual Space Learning with variational Autoencoders, if the classifier used for scoring is biased or inaccurate for specific attributes, the resulting performance metrics will reflect those errors rather than the true quality of the generative output. If the judge is flawed, the verdict is meaningless.
Architectural Sensitivity and the Baseline Problem
The most immediate issue with proxy-based evaluation is its fragility. Because these metrics depend on the internal states of an external model, slight changes to the proxy's architecture or training state can completely shift the evaluation baseline, making cross-paper comparisons nearly impossible.
Zhiming Zhou et al. exposed this vulnerability in Activation Maximization Generative Adversarial Nets, demonstrating that widely used metrics like the Inception Score are highly sensitive to the exact training state and architecture of the underlying classifier. This sensitivity prevents the establishment of a universal baseline across different research environments.
This problem extends directly to video and multimodal generation. In Temporal Shift GAN for Large Scale Video Generation, Andrés Muñoz 0002 et al. show that evaluation metrics utilizing auxiliary classifiers are frequently biased by the specific training history of those classifiers. Furthermore, when the proxy model's domain coverage is uneven, the metric becomes skewed. For example, Jibin Song et al. note in Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers that if a proxy model used to measure audio-visual synchronization is inherently better at identifying certain sounds or actions over others, the resulting synchronization scores will be fundamentally compromised.
The Circularity of Auditing Fairness and Bias
The danger of proxy dependency compounds dramatically when researchers attempt to audit generative models for societal bias or fairness. Using a secondary model to audit a primary model risks creating a closed, circular loop of distorted measurements.
[Generative Model Output] ---> [Proxy Model / Auditor] ---> [Skewed Fairness Metric]
^ |
|_________________________________|
Proxy's internal biases
corrupt the evaluation
Abhishek Mandal et al. document this dynamic in Generated Bias: Auditing Internal Bias Dynamics of Text-to-Image Generative Models. They show that auditing the internal states of text-to-image models requires secondary vision-language models to interpret latent representations; however, if these secondary models possess their own inherent biases, the resulting measurements become circular and unreliable.
This can lead to dangerous false positives in bias mitigation. Yusuke Hirota et al. discovered in Quantifying Societal Bias Amplification in Image Captioning that popular bias-mitigation techniques can satisfy an auxiliary classifier's narrow criteria while actually making the rest of the generated sentence more stereotypical.
When the evaluation classifier itself suffers from demographic skews or high error rates, the entire fairness benchmark collapses. As Mike Laszkiewicz et al. demonstrate in Benchmarking the Fairness of Image Upsampling Methods, relying on external classification systems to categorize demographic outputs introduces a severe dependency on the classifier's own accuracy. This blind spot is echoed by Jordan Vice et al. in On the fairness, diversity and reliability of text-to-image generative models and Md Mehrab Tanjim et al. in Discovering and Mitigating Biases in CLIP-based Image Editing, who argue that external evaluators and attribute classifiers often possess blind spots that misrepresent the true extent of identity drift and bias reduction.
Domain-Specific Proxies and Flawed Ground Truths
This dependency is not unique to computer vision and natural language processing; it spans structural biology, spatial simulation, and causal inference, where true ground-truth data is exceptionally difficult to obtain.
In biochemical generation, frameworks often rely on external folding and docking algorithms for both training signals and evaluation metrics. As Runze Ma et al. explain in RiboFlow: Conditional De Novo RNA Co-Design via Synergistic Flow Matching, the performance and metrics of the generative model are inherently bounded by the precision and potential biases of these third-party predictive tools. Pavel Strashnov et al. reinforce this in GeomMotif: A Benchmark for Arbitrary Geometric Preservation in Protein Generation, noting that errors or limitations in secondary validation algorithms can completely obscure the true performance of the generative model being tested.
Similarly, in camera-centric spatial generation, Kang Liao argues in Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation that evaluating spatially controlled images depends heavily on offline vision-based calibration models. Because these proxy models possess their own estimation errors, achieving an objective measurement of spatial simulation accuracy remains out of reach.
The problem becomes even more acute when evaluating counterfactual image generation—where "what-if" scenarios must be evaluated without any real-world ground truth image. To evaluate these models, researchers use "pseudo-oracle" classifiers to verify if an attribute change was successful. Work by Miguel Monteiro et al. in Measuring axiomatic soundness of counterfactual image models and Nefeli Gkouti et al. in Benchmarking Counterfactual Image Generation demonstrates that if these proxy classifiers possess low precision or inherent biases, the assessment of whether a generative model successfully implemented a causal change becomes fundamentally flawed.
The Fragility of Semantic and Uncertainty Judgments
In text generation, evaluating hallucinations, semantic agreement, or internal model uncertainty frequently requires an external model to judge whether two sentences mean the same thing.
Yixin Wan et al. (Sequence-Level Certainty Reduces Hallucination In Knowledge-Grounded Dialogue Generation) and Lukas Aichberger et al. (Improving Uncertainty Estimation through Semantically Diverse Language Generation) point out that semantic agreement and uncertainty estimation techniques rely heavily on the accuracy and robustness of external Natural Language Inference (NLI) models. If the underlying NLI model fails to correctly identify entailment between candidates, the resulting certainty scores will be completely unreliable.
When scaling this to generative rewards—where an LLM acts as an automated judge to score reinforcement learning pathways—the system's autonomy is constrained. As Mengzhao Jia et al. note in AutoRubric-R1V: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning, this reliance introduces potential biases or errors inherent to the judge model's own understanding. Furthermore, Weijie Tu et al. show in What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions? that using auxiliary models like VLMs to estimate class distributions on out-of-distribution (OOD) data introduces a severe dependency on the auxiliary model's own domain coverage; if the reference model performs poorly on an OOD task, the resulting performance rankings for all other models will be inaccurate.
Breaking the Dependency Loop
To escape the biased, resource-heavy, and circular feedback loops imposed by proxy-dependent metrics, a growing branch of research focuses on developing domain-agnostic, intrinsic evaluation methods.
Instead of relying on external pre-trained classifiers to evaluate model states, the goal is to monitor model convergence mathematically from the training process itself. A prime example of this direction is detailed by Sahil Sidheekh et al. in On Characterizing GAN Convergence Through Proximal Duality Gap. By introducing the Proximal Duality Gap, the authors provide a theoretically grounded, domain-agnostic metric that monitors GAN convergence intrinsically. Because it bypasses external classifiers entirely, it eliminates external bias and heavy computational overhead, pointing toward a future where models can be evaluated objectively without relying on an imperfect digital judge.
Key Foundational Work
Dual Space Learning with variational Autoencoders (2019) — contributed 1 concept elements to this direction.
Activation Maximization Generative Adversarial Nets (2018) — contributed 1 concept elements to this direction.