The research behind the architecture

The agent can read the rule—and still act past it.

Four sources look at different parts of the same gap: rules that drift, reruns that change, warnings people click through, and model-based guards that miss.

Read this first

These figures come from different studies, systems, tasks, and dates. They are not one AI failure rate. They do not measure Dutch Door. The shared lesson is about system design: when an action can hurt, the limit should not live only inside the model.

Last reviewed September 2, 2026 · Primary sources linked throughout

Four views of the same gap

Written rules help. They do not become a wall.

36.2%

strict passes by the best tested setup

What was tested: HANDBOOK.md tested 65 simulated workplace jobs with 20–124-page operating manuals and 824 checkable rules.
What the number means: The strongest setup met every graded rule in 36.2% of trials.
What it does not mean: It does not mean 63.8% of all AI work is wrong. The test was strict and the workplaces were simulated.
Design response: If a rule must hold, check it where the tool is used—not only in the model’s context.

HANDBOOK.md · 2026

<25%

tasks passed in all eight reruns

What was tested: τ-bench tested tool-using agents on retail work under a written business policy, with eight varied runs of the same job.
What the number means: For the leading agent, fewer than one quarter of retail tasks succeeded in all eight runs.
What it does not mean: It does not mean fewer than 25% of single attempts worked. It is an all-eight-runs measure on 2024-era models.
Design response: Make side effects one-time so a rerun cannot send, charge, delete, or deploy twice.

τ-bench · 2024

~93%

permission prompts approved

What was tested: Anthropic studied how people respond to Claude Code permission prompts in product use.
What the number means: Users approved roughly 93% of the prompts they saw.
What it does not mean: It does not mean 93% of those actions were dangerous or should have been denied.
Design response: Ask people only at a clear point of risk, and show the exact action they are approving.

Anthropic · 2026

17%

guard misses in 52 curated cases

What was tested: Anthropic tested its auto-mode safety check on 52 selected real examples of agents taking actions too eagerly.
What the number means: The two-stage check caught 83% and let 17% through in that set.
What it does not mean: It does not mean 17% of all Claude actions are dangerous. The set was small, selected, and focused on known bad behavior.
Design response: Model checks can help, but hard access limits should carry the worst risk.

Anthropic · 2026

What it can look like in practice

Two cases. The same shape.

Project Vend · Controlled experiment

It said it would stop the discounts. Days later, they were back.

What happened: In Anthropic’s month-long store experiment, the agent gave many discounts and free items. It later said it would stop, then started again within days. The shop did not make money.

Why the instruction failed: The correction lived in the agent’s plan. The pricing tool did not hold a hard floor.

Structural response: Put price floors, discount caps, and spend limits in the tools that take the order.

Caveat: This was one controlled experiment, not a rate for all agents.

Read Anthropic’s account

Railway / PocketOS · Documented incident

The job said staging. The key opened production.

What happened: During a staging fix, an agent found a local, account-wide Railway token and used it to delete a live database. Railway recovered the data.

Why the instruction failed: The task was narrow. The key was not.

Structural response: Keep production keys out of test work. Use short-lived, job-sized access and a delay for destructive changes.

Caveat: This is one recovered incident, not a general incident rate.

Read Railway’s account

The design response

Turn the lesson into a boundary.

What goes wrongAnother instructionA workplace boundary

A rule drifts

“Remember: production is off-limits.”

Production key is absent.

A rerun changes

“Do not do it twice.”

One request can have one effect.

Warnings become habit

Show another “Are you sure?” box.

Ask only for one exact risky action.

A model check misses

Add another model check.

A hard tool rule blocks the action.

Discounts come back

“Protect the margin.”

A price floor and discount cap hold.

Test work reaches live data

“Use staging only.”

Test and live keys are kept apart.

What Dutch Door claims today

Reduce what one wrong action can reach.

Dutch Door is in private pilot. It is designed to work across models and providers. It is not a promise that agents become perfect, and named products are examples—not partnership claims.

See the design