Why Proxy Metrics Fail Downstream Geometric Tasks

18 concepts8 papers1 proposal

When building complex computer vision systems, it is standard practice to break them down into modular components. We train an image retriever to find similar viewpoints, a local descriptor to match keypoints, or a landmark detector to track specific structural features. To evaluate these components, we rely on isolated proxy metrics: Recall@k for retrieval, patch-matching accuracy for descriptors, and Euclidean pixel distance for landmark detection.

The underlying assumption is intuitive: improving a component's proxy metric should inherently improve the performance of the end-to-end downstream application.

However, a growing body of research demonstrates that this assumption frequently breaks down. A model that dominates a patch-matching leaderboard or achieves near-zero landmark error often underperforms—or even fails—when integrated into real-world pipelines like 3D reconstruction, camera localization, or expression recognition. This disconnect is known as metric-task misalignment. It occurs because proxy metrics optimize for isolated, static properties, whereas downstream tasks depend on complex geometric relationships, structural connectivity, and functional utility.


Semantic Similarity vs. Geometric Alignment

The misalignment problem is highly visible in visual localization, where the goal is to determine a camera's precise 6-Degrees-of-Freedom (6DoF) pose. Most localization pipelines begin with an image retrieval step to find database images captured near the query image's location.

Traditionally, these retrieval models are evaluated using Recall@k—the probability that the correct ground-truth image or landmark is within the top $k$ retrieved results.

Standard Retrieval (Optimizes for Recall@k):
Query Image  ──>  [Matches Semantic Category]  ──>  Success! (High Recall)
(Ignores viewpoint changes, lighting, or spatial overlap needed for geometry)

Geometric Retrieval (Optimizes for Downstream Pose):
Query Image  ──>  [Matches Exact Spatial Overlap] ──>  Success! (Low Pose Error)
(Ensures enough shared keypoints exist to compute a 3D triangulation)

As shown by Pion et al. (2020) and Cabon et al. (2022), a high Recall@k score does not guarantee accurate downstream pose estimation. The root cause lies in a fundamental conflict between training objectives:

  1. Semantic Retrieval Objectives: Standard retrieval models are trained for landmark identification (e.g., recognizing a specific building). They incentivize viewpoint invariance—the model is rewarded for recognizing the building whether the photo was taken from the front, the side, close up, or far away.
  2. Geometric Localization Requirements: To calculate a precise camera pose via triangulation or relative pose estimation, the retrieved image must share a specific spatial relationship and sufficient co-visible area with the query image.

If a retrieval model returns an image of the correct building but from an entirely different perspective or distance, its Recall@k score is perfect, but the downstream geometric verification will fail. As Sattler et al. (2017) demonstrated, achieving sub-meter navigation precision requires moving beyond simple landmark recognition and designing retrieval mechanisms explicitly tailored to handle on-the-fly local 3D reconstructions.


Local Patches vs. Global Scene Connectivity

A similar disconnect occurs at the level of local feature descriptors (e.g., SIFT or deep-learned alternatives). For years, the computer vision community evaluated descriptors using isolated patch-matching benchmarks, which measure whether a descriptor can correctly match small, cropped image patches under artificial rotations or illumination changes.

When deep-learned descriptors emerged, they easily outperformed traditional hand-crafted methods on these patch benchmarks. Yet, when deployed inside end-to-end Structure-from-Motion (SfM) or 3D reconstruction pipelines, these learned models often performed worse than traditional methods like SIFT.

In a comprehensive evaluation, Schönberger et al. (2017) exposed why high patch-matching recall fails to translate to 3D reconstruction success. An end-to-end SfM pipeline does not just need a high volume of easy matches; it requires the ability to resolve highly ambiguous, difficult matches that act as "bridges" connecting disparate parts of a complex scene. Learned descriptors optimized purely on isolated patches often lack the global robustness required to maintain scene connectivity without introducing catastrophic geometric errors.

To bridge this gap, Luo et al. (2018) introduced GeoDesc, showing that local descriptors must be trained by integrating multi-view geometric constraints directly into the learning process, forcing the descriptor to respect the downstream requirements of the SfM pipeline rather than just optimizing for isolated patch classification.


Pixel Precision vs. Functional Utility

The insufficiency of proxy metrics extends beyond spatial navigation into tasks like facial analysis and object manipulation. Here, the misalignment manifests as a conflict between raw pixel-level precision and the functional requirements of a task.

Facial Alignment and Expression Recognition

In facial analysis, landmark detectors are traditionally evaluated using Mean Error or Euclidean distance relative to human-annotated ground truth points (e.g., the corners of the eyes or mouth).

However, optimizing solely for pixel-level landmark accuracy can degrade downstream performance. Belmonte et al. (2023) discovered that the most precise landmark detectors (by pixel distance) are often poorly suited for Facial Expression Recognition (FER). The reason is that standard geometric metrics prioritize static spatial precision but ignore temporal stability and functional relevance. A detector might achieve low average pixel error while introducing high-frequency jitter or micro-shifts across video frames, which the downstream FER model misinterprets as emotional expressions.

Proxy Metric Focus (Euclidean Error):
Minimizes absolute distance to manual annotations. Can introduce high-frequency jitter.

Downstream Task Focus (Expression/Pose):
Prioritizes structural stability and semantic consistency across frames.

Because human annotations are inherently noisy, treating them as absolute truth can bias models away from structural utility. To bypass this entirely, Chang et al. (2017) proposed FacePoseNet, a landmark-free approach that directly regresses 6DoF head pose. By evaluating alignment quality based on its direct impact on downstream face recognition rather than intermediate landmark errors, the authors demonstrated that bypassing the proxy metric altogether yields faster, more robust downstream performance.

3D Mesh Completeness vs. Object Pose Estimation

In robotics and augmented reality, 3D reconstructions of objects are typically evaluated using geometric metrics like mesh completeness or surface accuracy (e.g., Chamfer Distance).

Intuitively, a visually perfect, highly complete 3D mesh should be ideal for tracking an object's position. However, Burde et al. (2025) revealed that these standard geometric metrics do not reliably correlate with downstream object pose estimation. A reconstructed 3D model can be visually flawed, full of holes, or missing entire surface sections according to standard metrics, yet still perform exceptionally well in a pose estimation pipeline. This is because a pose estimator does not require a visually pristine surface; it only requires a sparse set of highly distinct, stable geometric landmarks to successfully calculate spatial orientation.


Designing for the End Goal

The recurring pattern across these diverse domains highlights a critical lesson for machine learning architecture: optimizing a component in isolation using a proxy metric often creates a blind spot to the structural and functional needs of the broader system.

Component Common Proxy Metric Downstream Task Why the Metric Misaligns
Image Retrieval Recall@k 6DoF Camera Localization Rewards viewpoint invariance; ignores the specific spatial overlap required for triangulation.
Local Descriptors Patch Matching Accuracy Structure-from-Motion (SfM) Captures easy, isolated matches; fails to measure the ability to resolve complex scene connectivity.
Landmark Detection Euclidean Pixel Distance Facial Expression Recognition Prioritizes static spatial precision; ignores temporal stability and functional relevance.
3D Reconstruction Mesh Completeness / Accuracy Object Pose Estimation Penalizes visual artifacts and missing surfaces that do not actually impede landmark-based alignment.

To build more robust vision systems, the field is shifting toward application-centric evaluation. Rather than asking how well a model performs on an isolated sub-task, researchers must evaluate models based on their end-to-end utility. This means training image retrievers to maximize geometric co-visibility, optimizing local features for global scene consistency, and evaluating structural detectors by their ability to stabilize downstream reasoning.

Key Foundational Work

FacePoseNet: Making a Case for Landmark-Free Face Alignment (2017) — contributed 4 concept elements to this direction.
Comparative Evaluation of Hand-Crafted and Learned Local Features (2017) — contributed 3 concept elements to this direction.
GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_retrieval_localization_metric_misalignme_c5b23d94,
  title = {retrieval-localization metric misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/retrieval-localization-metric-misalignment/rdr_c5b23d94c6714fa282b60a5b980f64e8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_retrieval_localization_metric_misalignme_c5b23d94_v1,
  title = {retrieval-localization metric misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/retrieval-localization-metric-misalignment/rdr_c5b23d94c6714fa282b60a5b980f64e8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake