Many complex computer vision tasks follow a intuitive "divide and conquer" strategy. To understand how a human interacts with an object, a system first detects the human and the object. To track an anomaly in a traffic video, it first localizes the moving vehicles. To segment a complex scientific image with high precision, it first generates rough region proposals.
This multi-stage, cascaded architecture is elegant because it breaks down a complex semantic problem into simpler, specialized sub-tasks. However, this design conceals a critical, structural vulnerability: the proposal-dependent recall bottleneck. If the initial stage fails to flag a region or object of interest, the downstream modules are completely blind to it. No matter how sophisticated the subsequent tracking, refinement, or multi-modal reasoning layers are, they cannot process what does not exist in their input stream. The recall of the entire system is strictly capped by the recall of the very first component.
The Mechanics of Cascading Failure
Why is this bottleneck so absolute? In a cascaded pipeline, the second stage operates entirely on the conditional space defined by the first stage. If the initial detector outputs a probability of zero for a true object, the downstream stage never receives the corresponding features or bounding boxes.
Consider a post-processing framework designed to evaluate prediction quality or refine boundaries. As Riedlinger et al. demonstrate in 3D Lidar object detection, meta-classification networks can filter out false positives or adjust confidence scores, but they are fundamentally incapable of recovering objects that the base detector missed entirely.
Similarly, spatial refinement techniques like the Local Response Context introduced by Schwartz et al. can clean up false alarms by analyzing neighboring confidence scores, but they cannot conjure a detection out of a completely blank response map. The downstream layers are mathematically isolated from the raw visual evidence that was discarded by the first stage.
[ Raw Input Image ]
│
▼
┌───────────────────────────┐
│ Stage 1: Proposal Gen. │ ──► [ Missed Object ] ──► (Lost Forever)
└───────────────────────────┘
│
▼ (Filtered Proposals Only)
┌───────────────────────────┐
│ Stage 2: Downstream Task │ ──► Blind to the missed object,
└───────────────────────────┘ regardless of model capacity.
The Blind Spot in Human-Object Interaction
The impact of this bottleneck is highly visible in Human-Object Interaction (HOI) detection. To identify an action like "reading a book," a two-stage framework first runs an object detector to locate the person and the book, then predicts the interaction verb connecting them.
In his foundational work on the HICO benchmark, Wang noted that errors in the initial object detection phase propagate directly through the system, acting as the primary bottleneck for higher-level scene understanding. If the detector fails to locate the book—perhaps due to occlusion or unusual lighting—the interaction model has no way to link the human to it.
As Chao et al. and Zhong et al. emphasize, this creates a hard performance ceiling. Even advanced frameworks designed to handle rare or zero-shot interactions, such as those proposed by Yu et al., remain entirely at the mercy of the initial region proposals. If the primary detector misses the entities, the semantic relationship is lost.
Downstream Collapse in Anomaly Detection and Tracking
The bottleneck becomes a safety-critical issue in dynamic environments like autonomous driving. Video Anomaly Detection (VAD) frameworks often use object detection as a preprocessing step to isolate actors before analyzing their behavior over time.
Zoellner et al. highlight how this dependency triggers catastrophic failures in adverse weather: if heavy rain or severe occlusion causes the initial detector to miss a vehicle, the downstream temporal analysis pipeline is starved of input, creating a dangerous blind spot.
Wang et al. observe a similar vulnerability in object-centric reconstruction frameworks; if an unconventional or small anomalous object is bypassed by the detector, the subsequent anomaly scoring layers never see it.
In tracking-by-detection systems, as Luiten et al. point out, if the proposal generator fails to identify a novel object early on, the tracking framework lacks the initial "seed" required to establish a trajectory, rendering long-term re-identification impossible.
The Limits of Foundation Models and Refinement
One might assume that powerful foundation models or interactive segmentors could bypass these limitations through sheer scale. However, empirical evidence shows they are subject to the exact same structural constraints.
When using the Segment Anything Model (SAM) for scientific imaging, Price et al. found that even state-of-the-art prompt-based refinement cannot recover objects that were completely omitted by the initial detection stage. The final segmentation quality remains strictly bounded by the base model's ability to localize candidates.
Similarly, in multi-modal vision-language models like LXMERT, Tan et al. demonstrate that relying on a frozen, pre-trained object detector introduces a fixed vocabulary and sensitivity ceiling. If the detector fails to flag a small or unusual object, the downstream cross-modality Transformer layers are forced to reason about an incomplete visual scene, leading to unrecoverable errors in cross-modal comprehension.
Conclusion
The proposal-dependent recall bottleneck highlights a fundamental truth about cascaded architectures: a pipeline is only as strong as its first filter. When design choices isolate downstream reasoning from the raw visual context, they trade maximum potential recall for computational efficiency and modular simplicity. Overcoming this bottleneck requires moving toward tighter coupling between stages—such as joint detection-and-tracking or end-to-end proposal-free architectures—ensuring that early errors do not become permanent blind spots.