Master 22 Solutions Blog

We ran an embedded evaluator on ourselves. Here is everything it found.

2026-09-19 · Paul Rodney Turner

ClawMaven checked 61 public claims against its own code. Only 17 were fully true. Here is what failed, why it happened, and how embedded evaluation now runs on every build.

On September 12, Dario Amodei published an essay proposing that outside evaluators be given permanent, employee-level access inside frontier AI labs. Not annual visits. Resident inspectors, the way bank supervisors sit inside the institutions they regulate. OpenAI said it would follow. Within days it was the most discussed idea in AI policy.

I build ClawMaven, a governance layer for AI agents. Our entire pitch is that we check whether what a company deployed matches what it says it deployed. So the obvious question was uncomfortable: what would an evaluator find if it walked into my company?

I decided to find out before anyone else did.

What we did

I pointed an AI coding agent at the ClawMaven repository with one instruction: read every claim the website makes about the product, then read the code, and report every place they disagree. Prices, feature lists, privacy statements, compliance mappings, the numbers we quote, the pages we serve to search engines, the files we serve to answer engines. Report only. Fix nothing.

It took about eight minutes and checked 61 claims across 14 surfaces. A second sweep of the forty pages it skipped the first time found more, including three pages that had been serving raw template code to Google where a price should have been.

The full table is public at clawmaven.com/claims-audit. Every finding, the file and line it lives on, what the code actually did, and what we changed.

What it found

Of the 61 original claims, 17 were true. The other 44 were wrong in one way or another: 14 flatly false, 9 stale, 7 overstated, 8 that said different things on different pages, 6 that could not be verified at all. Fifteen were severe enough that a paying customer would have been misled about price, privacy, or a feature they were paying for.

A few of them I want to describe properly, because the pattern matters more than the count.

The privacy page said we never collect your agent configuration. The homepage went further and invited readers to open their browser's network tab, run the wizard, and watch zero requests leave with their data. That check would have failed. An AI-assist feature was firing automatically on every pack generation, sending the project name, autonomy level and data sensitivity to a third-party model provider. Nobody had lied. The privacy page was true the day it was written. Then a feature shipped beside it, and nobody re-read the page.

The Team tier listed single sign-on. There was no single sign-on. There was a login dialog labelled SSO, with an Okta and Google Workspace picker, and an endpoint that accepted the provider name and threw it away, then checked a ClawMaven access token. A buyer could have demoed it as SSO in a sales call. That is worse than the feature being missing.

The site said you could cancel any time from your account dashboard. There was no account dashboard and no cancel endpoint. That claim applied to a $497 a month tier we had launched the day before.

The code itself had the same disease. The site said a governance pack was seven files; the real pack was seventy-nine. The verification script we ship with every pack promised to confirm nothing had been edited after delivery. It checked that the files existed. It never hashed anything. A flag called "enabled" meant approved in one file and blocked in another, so generated configs for three runtimes were inverted. And the word "signed" appeared on pages where the only signature was one only I could verify.

The rest were smaller doses of the same thing. ISO 27001 listed as a mapped framework with zero mapping code. Our audit service priced at four different numbers across five pages. Statistics on the audit page out of date in both directions. Even the audit's own summary paragraph miscounted its own table, which we discovered when we published it and derived the totals from the rows instead.

Why it happened

Not carelessness, or at least not the interesting kind. We had a file in the codebase whose whole job was to hold the real platform numbers, derived from the source. Every claim that used it was correct. Every wrong number was a literal someone had typed next to the feature instead of importing from the source of truth. The prose version of the same mistake produced the privacy findings: "browser-local" was written once and never re-derived as server endpoints grew up around it.

There was one more cause. Eight tests in the suite had been red for months. They were stale tests trailing correct code, harmless individually. But a suite that is always slightly red trains you to stop reading it, and that is how six wrong numbers on a public page survived. Fixing one of those eight un-hid sixteen more tests that had not run in months.

What we did about it

Four days, in order of how much a customer would care.

AI assist is now opt-in, with the disclosure sitting beside the button. The default wizard path is proven to make zero network requests by a test that stubs every outbound channel and fails naming the leaking URL. The privacy page now lists every flow that sends anything, what it sends, and to whom. Analytics loads only after consent. A fallback model provider we had never disclosed is gone from the chain. Cancellation goes through a real billing portal. SSO is gone from every page, and the endpoint pretending to be it is renamed to what it actually does. ISO 27001 is gone. The five frameworks we do map are now a single list derived from the mapping code itself.

The verification script now recomputes the SHA-256 of every file in the pack and fails on any mismatch. The flag means one thing. "Signed" appears only where a signature exists.

Every platform number, price and framework list on every surface imports from one file, and a test fetches each page the way a search engine does and fails the build if any served claim disagrees with the source. The sample pack prospects download is now generated by the real generator at build time, because the old one advertised a file structure the product never produced.

Version 1.10.6. Sixteen releases in four days, most of them removing things.

The point

A claim without a test is a wish. That is the whole finding.

Embedded evaluation is being discussed as a frontier-lab issue, five companies and a handful of institutions with the standing to sit inside them. But the discipline underneath it has nothing to do with scale. It is the willingness to let something outside your own head read your code against your words, and to publish what it finds rather than quietly patch it.

Every company shipping AI agents is going to be asked, by a customer, an insurer or a regulator, whether what they deployed matches what they said. Most will answer from memory. Memory is what produced 44 wrong claims on a site I read every day.

We are keeping the evaluator. It runs on every build now. If you want the same thing run on your agents, monthly, by someone who is not you, that is what ClawMaven Observer is.

The report is at clawmaven.com/claims-audit. Read it before you decide whether to trust the rest of the site.