Consent Triage Boards: Hierarchical Escalation for Revocable Data Permissions

Dynamic consent means letting people grant, narrow, and later revoke permission to use their data as its uses change. It is widely endorsed and rarely enforced. Web measurements find that 74.2% of sites never tell third parties about a revocation they had earlier reported as an acceptance. Language models, which are increasingly asked to decide whether a given data flow is allowed, leak sensitive information in a quarter or more of realistic tasks even when they answer privacy quizzes almost perfectly. This proposal borrows the structure of a journal editorial board, where an editor-in-chief sets policy, section editors judge within a domain, and reviewers do routine checks. We use it to build a triage system for consent decisions. Routine requests are settled instantly by a deterministic check against each person's consent record. Requests about new secondary purposes go to a short, domain-specific model judge. Only cases where that judge is uncertain go up to a top tier, a human or a stronger model. Each top-tier ruling is written back to the lower tier as a precedent.

The claim is narrow and testable. The editorial hierarchy does not help because it adds agents. It helps because it separates questions that have a definite answer ("has this person revoked this scope?") from questions that need judgment ("is this new research purpose compatible with what they agreed to?"). It also spends expensive review only on the second kind. We expect this arrangement to reduce consent violations and over-refusals relative to one model that is given the whole consent history. We expect it to do so at a controlled rate of escalation and with more stable decisions across prompt formats. We build a benchmark that attaches consent timelines to existing privacy tasks and test that claim against strong single-model and compute-matched baselines.

What exists and what is missing

Two groups of papers motivate the problem. The first shows that consent breaks down in practice. Johnny Can't Revoke Consent Either crawls websites and finds three failures. First, 57.5% of sites keep processing advertising and analytics cookies after revocation. Second, 14–16% of sites built on commercial consent platforms still store a positive consent flag after the user revokes. Third, revocation rarely reaches downstream recipients. Ethical Considerations for Responsible Data Curation argues that consent must be designed in before collection, not patched afterwards. It proposes data consortia and registered reports as institutional oversight, but specifies no mechanism for them. Qualitative work with creative professionals (Governance of Generative AI in Creative Work) finds that people reject one-time consent for AI training. The internet of things: interaction challenges to meaningful consent at scale warns that constant prompting produces reflexive acceptance. It suggests treating consent as an asynchronous review task handled by agents, which is the role our lower tiers take on.

The second group shows that model judges are unreliable on exactly these decisions. PrivacyLens tests models on agent tasks in simulated Gmail, Slack, and Notion tools. It reports that GPT-4 answers 98.99% of privacy probing questions correctly but leaks sensitive data in 25.68% of agent actions. Smaller models leak less by becoming uselessly paranoid. Can LLMs Keep a Secret? builds the ConfAIde benchmark, where GPT-4 leaks secrets 39% of the time in a meeting-summary task. Both papers use contextual integrity, the theory that privacy means information flowing according to the norms of its context. Contextual Integrity in LLMs via Reasoning and Reinforcement Learning trains models to reason explicitly about sender, recipient, and purpose, using only 729 synthetic examples, and cuts PrivacyLens leakage by up to 40%. That method is our strongest single-model competitor, and we reuse its training data. Policy-as-Prompt constrains our design in two ways. It finds that verbose policy prompts perform worst. It also finds that prompt variants with similar average accuracy (~0.79) disagree on edge cases, which it calls predictive multiplicity. Together these findings argue against one judge holding a long rulebook and argue for short, domain-scoped policies.

On the governance side, Embodied LLM Agents Learn to Cooperate in Organized Teams shows that leader–worker structure and a critic layer improve embodied teamwork, with a 9.76% faster completion time for three GPT-3.5 agents. That evidence concerns household coordination, not information governance, so it tells us little about whether hierarchy helps consent decisions. ARGO describes a three-tier organizational structure: a shared policy foundation, a central tooling layer, and local implementation. It has no consent semantics. Usage Governance Advisor governs pre-deployment choices of which model to use for which task, using a knowledge graph and a model judge. It does not model revocation. None of these papers treats a person's changing consent record as the object being governed. None measures the trade-off among errors, escalations, and latency, which Gap 2 of the research file identifies as the central tension.

Core Claim

A three-tier consent board beats a single model at a matched compute budget. The board has a deterministic record check at the bottom, short domain-scoped judges in the middle, and uncertainty-triggered escalation with precedent write-back at the top. It should achieve lower consent-violation rates at equal or better helpfulness, fewer decision flips under prompt reformatting, and complete revocation propagation. It should reach this operating point with a small and declining share of cases escalated. If a single model with the consent record in its context matches the board, the hierarchy is unnecessary and we will report that.

Research Program

The central artifact is a benchmark we call Consent Timelines, because no dynamic-consent benchmark exists in the file. Each item takes a seed scenario from PrivacyLens agent tasks or ConfAIde Tier 2–3 vignettes. It attaches a per-person consent history: which data categories were granted, for which purposes, and to which recipients, followed by later narrowing or revocation events. The item ends in a request that the system must allow, deny, or partially allow, for example by redacting fields. Requests come in three strata. In record-decidable items, the answer follows mechanically from the history, such as a use after revocation. In purpose-compatibility items, a new secondary purpose must be judged against the original grant. In multi-party items, the data concerns several people with different consent states. Record-decidable labels are computed by rule. The other two strata are labelled by several human annotators, and their agreement is reported. Agreement sets a ceiling on achievable accuracy.

Each item also includes a small downstream graph of recipients who previously received the data. This lets us score whether a revocation actually reaches every one of them. The measure is the operational analogue of the 74.2% failure found on the web.

The board has three tiers. The bottom tier compiles consent histories into a state machine and resolves record-decidable questions without any model call. The middle tier is a set of domain judges. Each is a model prompted with a short, structured policy for one domain, such as health, education, or workplace data, following the Policy-as-Prompt finding that structured prompts work best. Each judge also has access to a case library of past precedents. We fine-tune these judges with the contextual-integrity reinforcement learning recipe on the HuggingFace contextualintegritysyntheticdataset (729 vignettes). A domain judge escalates when its confidence falls below a threshold. Confidence is estimated from agreement across sampled judgments and across prompt formats, so predictive multiplicity becomes the signal for escalation rather than an unnoticed source of noise. The top tier is a stronger model or, on a subset, a human annotator. Its rulings enter the case library.

We choose the threshold $\tau$ to minimize

$$C(\tau) = E(\tau) + \lambda \, R(\tau),$$

where $E(\tau)$ is the rate of wrong decisions on non-escalated items plus top-tier errors on escalated ones, $R(\tau)$ is the fraction of items escalated, and $\lambda$ is the cost of one escalation relative to one error. Sweeping $\lambda$ traces out an error–escalation curve, and the full curves are what we compare across systems.

Baselines

The naive reference is broad consent with one-time approval. Every request that falls within the original grant's category is allowed, and later revocations and purpose changes are ignored. It shows how much the benchmark actually depends on consent dynamics. If it scores well, the benchmark is too easy.

The strongest existing method is a single judge trained with contextual-integrity reinforcement learning, as in Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. We use the same base model as our domain judges, Qwen2.5-7B, and the same 729 training examples. The judge receives the full consent history and one combined policy in its context. That paper reports up to a 40% reduction in PrivacyLens leakage relative to base models, and reports that its 7B model outperformed a 14B baseline on integrity scores. For reference, frontier models without this training leak in 25.68% of PrivacyLens actions (GPT-4, PrivacyLens) and 39% in ConfAIde Tier 4 (GPT-4, Can LLMs Keep a Secret?). Those numbers are for the original benchmarks. No reported result was found on Lacuna for any method on consent-timeline tasks, so on our benchmark every baseline number will be new. We also run GPT-4-class single judges given the same consent history.

To be fair, every system gets the same training data, the same hyperparameter-search budget, and the same total inference token budget per item. For the board, the budget counts escalations to the top tier. A single model given the same compute may use it for self-consistency voting.

Beating the strongest baseline means the following. At a matched budget, the board's error–escalation curve lies below the single judge's error rate, where escalation is not available to the single judge. On purpose-compatibility and multi-party items, the board shows lower violation rates without higher over-refusal. And the board achieves this with at most roughly 15% of items escalated.

The ablations isolate each ingredient. Removing the deterministic bottom tier tests whether routing record questions away from the model matters. Merging all domain judges into one judge with the combined policy tests domain scoping. Replacing uncertainty-triggered escalation with random escalation at the same rate tests whether the uncertainty signal carries information. Disabling precedent write-back tests whether escalations become rarer over a stream of requests.

Why not just…?

Why not just give one strong model the consent record and the contextual-integrity policy? This is the obvious competitor and the reason the CI-trained single judge is our main baseline. The evidence against it is indirect. The gap between knowing privacy norms and acting on them persists even with privacy-enhancing prompts, which still leak in over 25% of cases (PrivacyLens). Long combined policies degrade accuracy and produce edge-case disagreement (Policy-as-Prompt). Neither finding was measured on consent histories, though. The single-judge baseline and the "merge domain judges" ablation settle the question directly.

Why not just add a deterministic access-control check in front of one model? This is the most serious threat. It may capture most of the gain, because many consent violations are simply uses after revocation. We therefore include it as an explicit ablation: bottom tier plus a single judge, with no domain scoping and no escalation. If it matches the full board on record-decidable items, that is expected. We then limit the claim to purpose-compatibility and multi-party items. If it matches the board there too, the hierarchy above the rule check adds nothing and the paper reports that result.

Why not just spend the same compute on self-critique or majority voting? Hierarchical gains in Embodied LLM Agents Learn to Cooperate in Organized Teams could reflect extra model calls rather than structure. Our compute-matched self-consistency baseline answers this. So does the random-escalation ablation: if random escalation at the same rate performs as well, the benefit comes from spending more on some items, not from knowing which items need it.

Why not just use a commercial consent-management platform? These platforms record consent on the client side. The measurements in Johnny Can't Revoke Consent Either show positive consent flags that persist after revocation and revocations that are not propagated to third parties. They also make no judgment about purpose compatibility, so they cover only our bottom tier, and imperfectly. The downstream propagation score measures this directly.

Decisive Checks

Three results would decide the proposal. The first is the error–escalation curve on purpose-compatibility and multi-party items, compared against the CI-trained single judge and the rule-check-plus-single-judge ablation at a matched budget. If neither curve falls below the board's, the core claim fails. The second is decision stability. We rephrase each item in plain text, JSON, and YAML formats and count how often the decision flips. We predict fewer flips for the board, because format disagreement triggers escalation instead of silently changing the answer. The third is the escalation rate over a stream of 2,000 requests, with and without precedent write-back. If precedents work, escalations should fall while accuracy on later, similar items rises. If the escalation rate stays flat, the top tier is only an expensive oracle and not a governance structure that improves the lower tiers.

We also report revocation propagation completeness on the downstream graphs. The bottom tier should reach 100% by construction, which makes this a sanity check rather than a finding. We report it because the web measurements show this property is not a given in deployed systems.