Enforcing Global Consistency on Local Predictions

21 concepts11 papers1 proposal

When deep learning models analyze complex, structured data—such as videos, long audio clips, or multi-step processes—they often operate hierarchically. They break the input down into local units: frames, pixels, short segments, or individual attributes. However, relying solely on these local features introduces a fundamental vulnerability: local ambiguity.

A single frame of a video showing a hand stirring a pot might look identical whether the person is making a hot dog or brewing tea. Without broader context, a frame-level classifier can easily output high-confidence predictions that are completely out of context or logically impossible given the overall activity.

To solve this, a powerful paradigm has emerged across computer vision, audio processing, and sequential modeling: hierarchical semantic filtering. Instead of treating local steps as isolated events, these methods use a bottom-up pass to infer a global context, and then use that global context as a top-down guardrail to filter, mask, or rectify local predictions.


The Core Mechanism: Top-Down Masking and Filtering

The intuition behind hierarchical semantic filtering is straightforward: the whole constrains the parts. By enforcing a strict relationship between global and local labels, models can prune the search space and eliminate false positives.

[ Local Features ] ---> [ Bottom-Up Aggregation ] ---> [ Global Context Prediction ]
        |                                                          |
        v                                                          v
[ Raw Local Predictions ] <--- [ Top-Down Semantic Filter ] <------+
        |
        v
[ Consistent Final Output ]

As Wang et al. demonstrate in large-scale video recognition, segment-level models frequently suffer from high false-positive rates because they predict concepts that contradict the broader semantic theme. To fix this, the authors enforce a subset relationship between local segment labels and global video labels. If a concept is not predicted at the video level, it is filtered out at the segment level, ensuring that short-term actions remain semantically aligned with the overall theme.

This top-down refinement can take several structural forms:

  • Soft Constraints: Xu et al. train a global classifier to identify all categories present in a video sequence. This global prior acts as a soft constraint on local pixel-level actor-action segmentation, pruning the optimizer's search space and preventing the model from assigning labels that are inconsistent with the video context.
  • Consistency Masking: In instructional video analysis, standard models often misidentify steps because they ignore the unified goal of the video. To counter this, Tang et al. (2019) and Tang et al. (2021) use an inferred global task label (e.g., "Changing a Car Tire") to apply a task-specific mask to local step scores. Any predicted action that is logically incompatible with the primary goal is suppressed.
  • Feature Bottlenecks: This concept extends beyond videos to explainable image classification. Panousis et al. introduce Coarse-to-Fine Concept Bottleneck Models, where high-level class predictions are used to mask and filter the available pool of low-level attributes. This ensures that fine-grained features are interpreted within the correct semantic context, preventing the model from identifying irrelevant or contradictory attributes.

By treating global context as a filter, the model doesn't need to learn every complex local-to-local transition rule perfectly; it simply uses the global category to eliminate noise. For instance, in audio processing, Guo et al. apply clip-level classification as a guardrail for frame-level sound event detection, effectively cleaning up localized noise and false positives. Similarly, in video object detection, Li et al. utilize global video-level statistics to suppress isolated, low-confidence misclassifications that lack contextual support across the entire sequence.


Dynamic and Recurrent Refinement

While static masking works well when the entire input is available upfront, many real-world applications require understanding sequences as they unfold, or iteratively cleaning up initial guesses.

When a sequence is processed over time, context accumulates. McCarthy et al. demonstrate in sequence-based predictive modeling that incorporating real-world data from the initial steps of a sequence provides critical context that drastically reduces error margins for subsequent predictions.

To capture this temporal flow, models can integrate global context recurrently. Ahn et al. developed a Hierarchical Action Segmentation Refiner (HASR) that uses a Gated Recurrent Unit (GRU) to combine initial local segment labels with global video context. By processing these inputs together, the module can look at the logical flow of the entire activity and actively correct out-of-context action predictions (such as predicting "stir" while making a hot dog).

[ Initial Segment Labels ] ---\
                               ===> [ GRU Refinement Module ] ---> [ Corrected Labels ]
[ Global Video Context   ] ---/

In offline or retrospective analysis, this refinement can even happen backwards in time. Chen et al. introduce a dynamic rectification mechanism for surgical phase localization. As a procedure progresses, the system leverages the relationship between early and late stages. By maintaining a dynamic sequence of results, it utilizes the emerging global context to retrospectively go back and correct earlier, less certain classifications.


Why It Works: Resolving Ambiguity

Hierarchical semantic filtering works because it resolves the fundamental limitation of local features: under-determination. A local patch of pixels, a single frame of video, or a brief window of audio does not contain enough information to guarantee a correct classification.

By structuring the network to enforce global-to-local consistency—whether through a topic node that ensures narrative coherence (Rohrbach et al.), a task mask (Tang et al.), or a subset constraint (Wang et al.)—the system injects domain knowledge into the architecture itself. The model no longer has to guess blindly based on noisy local data; it uses the global context as a blueprint to decide what is possible.

Key Foundational Work

Actor-Action Semantic Segmentation with Grouping Process Models (2016) — contributed 2 concept elements to this direction.
Coherent Multi-sentence Video Description with Variable Level of Detail (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_semantic_filtering_614eae4e,
  title = {hierarchical semantic filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-semantic-filtering/rdr_614eae4e087e4690b5b0505e057b4cd5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_semantic_filtering_614eae4e_v1,
  title = {hierarchical semantic filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-semantic-filtering/rdr_614eae4e087e4690b5b0505e057b4cd5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake