Human Approval Is a Privilege Escalation Path

An agent wants to refund $200. A card appears, someone reads it, clicks Approve, and the refund goes out. The log records that a human approved it.

Two questions decide whether that record is worth anything. The first one gets asked a lot: what did they approve; the sentence on the card, or the request that eventually hit the API? The second one almost never gets asked, and it is the one that turns a safety feature into a hole:

What does your system do with the word "approved"?

If the answer is "it skips the check," you haven't added an approval step. You've added a way to get past the check, and you've put the button in front of a person who has no way to know that's what they're holding.

The obvious implementation

Here is what nearly everyone writes, because it is what the feature sounds like:

# The agent asked to run something risky. A person said yes. So run it.
if approval.status == "approved":
    result = await tool.run(**approval.arguments)

Read it as an attacker would. The check that would normally decide whether this call is permitted is not in this code path at all. approved is not satisfying a condition, it is bypassing the place where conditions get evaluated.

So the exploit writes itself. Find an action the agent is never allowed to take. Get it in front of a human; social engineering, an ambiguous summary, a prompt injection that makes the model request something plausible-sounding. The human, who sees "the assistant wants to do X, approve?", clicks yes because that is what the button is for. And now their click is the authorization. Not a confirmation of an authorization that already existed, the authorization itself.

The people clicking these buttons all day are support staff. They are not adjudicating whether an action is permitted by policy; they assume the system already did that, and that they are being asked a business question. They are right to assume it. The system is what's lying.

Three outcomes, not two

The fix is not more validation around the approval. It's that a permission check has to have somewhere for approval to fit, rather than a place for it to jump over.

A pending call resolves to one of three things:

PRECEDENCE = ("deny", "require_approval", "allow")   # most restrictive first

def evaluate(rules, *, tool, source, arguments):
    matched = [r for r in rules if r.matches(tool, source, arguments)]
    if not matched:
        return ALLOW
    for effect in PRECEDENCE:
        winner = next((r for r in matched if r.effect == effect), None)
        if winner is not None:
            return Decision(effect, rule=winner.name)
    return ALLOW

Two properties matter here and both are load-bearing.

Combination is order-independent. Whoever authored the rules did not have to think about ordering, because the outcome is the same however the list is shuffled. A policy layer whose behaviour depends on rule order is a hand-written if/else chain wearing a costume, and it will eventually be edited by someone who doesn't know the order matters.

An explicit allow cannot overturn a deny. This is the same stance IAM and Cedar take, and for the same reason: a carve-out is for narrowing something permitted, never for punching through something forbidden.

Now approval has a place to fit; the middle one and only the middle one:

decision = evaluate(rules, tool=name, source=source_id, arguments=args)

if decision.effect == "deny":
    # `approved` is not consulted. There is nothing here for it to spend.
    raise Blocked(decision)

if decision.effect == "require_approval" and not approved:
    hold(name, args, decision)
    raise HeldForApproval(decision)

return await adapter.execute(name, args)

The whole claim of the design is in what that first branch does not read. An approval satisfies require_approval and nothing else. If a rule denies the call, a human's yes is not a stronger form of permission that overrides it; it is simply not the question being asked.

Because precedence puts deny above require_approval, a denied call never becomes an approval request in the first place. That has a nice second-order effect on the person reviewing: everything in their queue is genuinely a decision someone authored for a human to make. They are never unknowingly rubber-stamping a misconfiguration.

The flag has to say which decision, not just that there was one

Take the "one lock" claim seriously and a second question falls out: which lock and who handed over the key?

We got this wrong at first. approved was a bare boolean, so it satisfied whatever require_approval rule happened to be asking at execution time, which is not necessarily the rule the person was asked about. Rails get edited. A rule that appears after a call is held is a question nobody has answered, and a bare boolean will answer it anyway.

There is a sharper version of the same bug if you have more than one kind of approval. Ours also asks the customer to confirm before an agent changes their own data, a different question, from a different person, with different authority. Those holds are raised with no rule attached. A bare boolean let one of them satisfy a staff rule that appeared while the customer was deciding.

So the flag carries the decision it came from:

answers_this = approved and approved_rule == decision.rule_name

if decision.blocked and not (decision.needs_approval and answers_this):
    ...

A customer confirmation carries None, which can never equal a named rule. A staff approval carries the rule it was raised under, and a rename or a new rule is a mismatch, so the call is held again, under whatever is asking now, and someone answers the question that is actually being asked.

The thing you approve has to be the thing that runs

The escalation hole is the one nobody talks about. This is the one everybody finds eventually, usually after an incident.

The approval was granted against something. If anything between the yes and the execution rebuilds the request, a data refresh, a recalculation, or the model being asked to try again, then the object the human saw and the object that ran are two different objects, and the log linking them is decoration.

The usual fix is to freeze the payload and hash it. That's right, but it leaves open the question that decides whether it works in practice: who replays it?

If the answer is "the agent," you have quietly reintroduced the problem. Telling the model try again, you're allowed now means it regenerates the call, and generation is not deterministic. Sometimes the arguments come back slightly different. Then you are diffing regenerated payloads against approved hashes and sending people back to re-approve trivial deltas, which trains them to click without reading, and a reviewer who clicks without reading is worse than no reviewer because now there's a signature on it.

So the server replays the stored call itself:

UPDATE tool_approvals
   SET executed_at = now()
 WHERE id = $1
   AND executed_at IS NULL
RETURNING tool_name, arguments_enc;

A few notes on this statement:

It is claimed before the call goes out, not after it comes back. A crash between the claim and the call leaves an approval marked spent and an action that may not have happened, recoverable and a human can look at it. The other ordering leaves an approval that can be spent twice, which is the failure you cannot see.

It is conditional on executed_at IS NULL, so two concurrent turns racing the same approval produce exactly one winner at the database rather than in application logic.

And the model is never asked twice, so there is no second generation to diff against anything.

The arguments are encrypted at rest with the tenant's key, like message bodies: whoever reviews an approval has to see them, and they contain customer data, so they may exist in a row and never in a log line.

Re-checked at execution, not just at hold time

The stored call does not get a fast lane back in. It goes through the same evaluate above, with the current rules, the only difference being that approved=True is now available to satisfy require_approval.

This matters because approval introduces a gap in time, and rules change during gaps. Somebody tightens a policy while a refund sits in a queue over lunch. The call comes back through the gate, the new rules deny it, and it does not run, even though a person approved it an hour ago under rules that permitted it.

That is the correct outcome and it is worth being explicit that it is a choice. The approval was a decision about a business question. It was never a promise that policy would hold still.

Two tests are worth writing, because both of these are things people assume rather than verify:

# 1. A policy change mid-run is picked up by the next call, without rebuilding anything
registry = build(rules=[allow_refunds])
assert (await registry.call("create_refund", {"amount": 500})).success
registry.policies = [deny_refunds]              # someone edited the rules
assert not (await registry.call("create_refund", {"amount": 500})).success

# 2. An approval cannot launder a deny however many times you replay it
registry = build(rules=[deny_refunds, approve_big_refunds])
for _ in range(3):
    result = await registry.call("create_refund", {"amount": 500}, approved=True)
    assert not result.success
assert adapter.calls == [] # it never reached the adapter

The second one is the whole post as an assertion. If it fails in your system, someone can get anything run by getting a person to click a button.

Both executions and refusals get a policy_version stamped on them, for us it's the timestamp of the rule set that decided. Without it, "the rules were edited at some point today" makes every record from today unfalsifiable.

What the reviewer is actually shown

A separate mistake, and one we do for a while: the card named the rule that held the call.

That sounds sufficient. It isn't, because a rule's name is free text somebody typed. "Refunds over $100" explains itself. "temp" explains nothing, and now the reviewer knows an action was stopped but not what the person who wrote the rule was worried about. So they evaluate the action on vibes; the customer seems fine, the amount seems normal which is exactly the judgment the rule existed to replace.

The information that answers it was being computed at the gate and thrown away: the condition that actually matched. Not Held by "temp" but:

amount (240) > 100

Now the reviewer knows which claim to check. Two details turned out to matter while implementing it. Only the deciding rule's conditions get reported, a weaker rule that also matched would otherwise put its own reasoning on the card and point the reviewer at the wrong thing. And the rendered condition quotes argument values, so it inherits the arguments' handling: encrypted in its own column, and specifically kept out of the audit event buffer, which is documented as carrying no user data and has to stay that way.

Where this stops working

Freezing the payload does not freeze the world. This is the sharpest objection to everything above and it came from someone in a thread, not from us:

You approve a $200 refund, someone in support refunds it manually in the gap, and replaying the exact approved payload is now a double refund.

The payload is byte-identical and the action is wrong. Re-evaluating policy doesn't catch it either, because no policy rule was violated, the world moved. What would catch it is recording the predicates the decision rested on (amount <= remaining_refundable, order still in the state it was) and re-evaluating those at execution. We record which conditions held, which is not the same thing: our conditions are the rule's tests, not the business facts the human was reasoning about. That gap is open.

A TTL helps and is worth having, ours expires held calls after 24 hours, read at access time rather than by a sweeper, so there's no background job that can silently die and leave a stale row executable. But be honest about what a TTL is: it bounds the window, it does not detect the change. The manual-refund case happens thirty seconds after the click. A 40 minute old approval on a settled order is fine.

And if the provider accepts the call but your process dies before recording the response, you cannot tell from your own logs whether it landed. The fix is an idempotency key generated and written down before the call, so recovery asks the provider what became of that key instead of interrogating logs that by definition weren't written. That only works where the provider supports idempotency keys, and for arbitrary tools it usually doesn't. The honest state after that crash is "unknown," and the expensive mistake is retrying on the assumption it never landed.

Two things to check in your own system

Print what your approval path skips. Find where the approved branch runs the action, and trace backwards to your permission check. If the check is not on that path, approved is a bypass, whatever the variable is called.

Then try to launder a deny. Construct a call your rules refuse outright. Force it into the approval queue, call the hold function directly if you have to. Approve it. If it runs, the button in your UI is a permission escalation that any of your staff can be talked into pressing, and the audit log will faithfully record that a human authorized it.

If that one passes, there's a follow-up that catches the subtler version: hold a call, then add a new rule that would stop it, then approve the old request. If it runs, your approvals are answering questions nobody was asked.

I work on CoreBase, a governance layer for agents that act on customer data. CoreMCP, our on-prem bridge, is open source.