Register
The environments, and what each one can prove
A verdict of gradable means the adversarial suite ran every named cheat against its own guard and the guard refused it. Graders reads expectations held over expectations declared. A row is a link to that environment's task book.
schema compound-evals/results@1read envs/*/results.jsonsuite adversarial/prove_graders.py
Register
Sorted by product name.
24 environmentsWhat the environments refuse to grade
Routes that look task-shaped and write nothing a guard can read.
190 recordedA route that writes no row cannot be graded by reading rows. These are named in each environment rather than dropped, so the task book states what it left out and why.
| Environment | Not gradable | Why it is out |
|---|---|---|
 agentwire7 recorded | POST /api/revalidate | the most task-shaped route in the app and it writes no row: revalidateTag plus revalidatePath, then return. src/lib/posts.ts records that on Cloudflare Workers the tag cache resolves to `dummy`, so since the 2026-09-16 move the tag half has been a no-op as well. |
 breachprobe10 recorded | POST /api/checkout | it posts to api.stripe.com to create a live checkout session. No Stripe key is set here and none will be, so the route answers 503 'Payments are not configured yet.' and writes nothing |
 cardchase5 recorded | pause recovery for a customer (cardchase_customers.do_not_contact) | the switch every gate in the product stops on, and its ONE writer is POST /api/stripe-app/action, which is gated on a Stripe app signature. The check is pure crypto with no network call, so a task could be built on it and would measure whether a model can forge an HMAC rather than whether it can operate the product. Same for connect and context in that directory. |
 clausewatch9 recorded | POST /api/checkout, GET /api/billing/portal, POST /api/webhooks/stripe | checkout creates a genuine Stripe subscription, the portal opens a hosted session, and the webhook verifies a signature before it writes. The first spends money on the estate's live account and the other two grade a forgery rather than the product |
 covercheck7 recorded | create and edit requirement profiles (server action) | the action is exported and correct and nothing imports it; the live product is one console at / and every /dashboard/<tab> path is a two-line redirect to it |
 fetchdue21 recorded | every write control in the console (/) | page.tsx renders its own NOT_WIRED sentence for a signed-in member and pressing Approve on a real row caused zero requests to /api/, measured 2026-09-19 in demo/console-views.txt. No task can be driven through the browser, so browser_tasks is 0. |
 frontwire7 recorded | POST /api/checkout | it calls the studio's live Stripe account and creates a real Checkout Session. The product's own README records an audit sweep clicking one on 2026-09-13 and creating a live session. Nothing in this environment touches it, and the build carries a placeholder key so the route answers 500 rather than reaching Stripe. |
 leadgrade12 recorded | GET /api/integrations/oauth/hubspot and its callback | connecting is the one writer of leadgrade_integrations and it needs a real authorization code exchanged against api.hubapi.com. Nothing here completes a third-party OAuth flow or holds a token. The fixture seeds the row the callback would have written, without a usable token, and the three rail paths are graded on what they do next. |
 matchline7 recorded | POST /api/checkout | the second half of the handler is a live Stripe Checkout Session create. MatchLine bills on the second live key on this machine (acct_1U94AeHX6skw2rM7, the one OutRip and WireCall use), and no Stripe call may be made here. With the key absent, which is the product's own not-configured path, the route inserts the ml_orders row and then throws, so the only outcome to grade is a 500. Measured 2026-09-19. |
 matchrail9 recorded | POST /api/receipts, the goods receipt | MEASURED BROKEN, and it is defect 1 below. Its upsert names matchrail_documents_source_external, which is a PARTIAL unique index, and PostgreSQL will not use a partial index for ON CONFLICT unless the statement repeats its predicate. Every call answers 400 'there is no unique or exclusion constraint matching the ON CONFLICT specification', so no receipt can be recorded through the product and there is no workflow to grade. Reproduced against the running app and again in raw SQL on 2026-09-19. |
 outrip8 recorded | POST /api/checkout | every call creates a live Stripe Checkout Session; there is no offline path and no test-mode fixture that does not spend a key. The rows it would write are seeded instead, and the tasks grade what happens after the money lands. |
 parserail7 recorded | the other 38 /v1/* capability endpoints | each ends in a model call. No inference key is set here and none will be: a score is only ever produced on a free rail, so spending ANTHROPIC_API_KEY, OPENAI_API_KEY or GEMINI_API_KEY to make one is out. Their not-configured path is what queue-the-manifest-parse grades instead. |
 popwire7 recorded | GET /api/digest-items | the most task-shaped route in the app and it writes no row: it reads the feed, renders the day's email through the production edge function and returns JSON. The send row is written by the lane's sender. |
 soft-money-journal5 recorded | the weekly letter, compound-ops/letters/send-letter.py | its only database effect is a publication_letter_sends row written AFTER a mail provider accepted a real message. --dry decides every recipient and writes no row, so the workflow cannot be graded without sending mail, which this environment may not do. |
 stacktab8 recorded | GET /api/catalogue, GET /api/estimate, GET /api/recommend, GET /api/services/[category], GET /api/freshness, GET /api/health | six of stacktab's seven route handlers write nothing at all. A grep for .insert( .upsert( .update( .delete( .rpc( over src/ and scripts/ returns exactly one line, the upsert in /api/watch. Rule 3 says every reward reads database rows, so a reward on a pure read would have to grade a JSON body or an HTTP status, which is what a broken write looks like from the outside. |
 standup4 recorded | follow a vendor and be alerted (standup_watchlists, standup_alert_sends) | both tables carry RLS, foreign keys, a check constraint and a unique index, and neither is referenced once in src/ or scripts/. Counting every standup_ token in the repo on 2026-09-19 returned 0 for both |
 starreply7 recorded | GET /api/cron/overnight, the nightly pass | runPass() starts by polling the rail. Both rails call live hosts (mybusiness.googleapis.com, api.trustpilot.com) with no base-url override, so offline the pass writes one poll event and returns. Grading it needs a stubbed rail, which would be grading a stub. |
 still-mornings8 recorded | compound-ops/letters/send-letter.py, the weekly letter | its only database write is the publication_letter_sends row appended AFTER compound_mail.send() returns ok, and its --dry path continues before both, so no path through it writes a row without putting real mail on the wire. Nothing here runs it. |
 thismuchweknow7 recorded | the archive sync into publication_posts, which is the third task on all four sibling publications | nothing writes it for this publication and nothing in this product reads it. compound-ops/social/ugc/publish.mjs's SITES list names four repos and not this one; production holds 0 rows for thismuchweknow in all three publication tables while the siblings hold 73, 73, 80 and 82 posts; and this tree has no src/lib/live.ts. Measured against the running build: with 0 rows the landing lists 5 entries, and with a row inserted for this publication it lists 5 entries and /entry/<that slug> answers 404. There is no entry point to run and no row to grade. scripts/up.sh re-checks both ends and fails closed if either changes. |
 triagedesk11 recorded | Every control on the product's own console: Approve, Edit, Kill and Take it back | `buildLiveConsole` sets each row's key to the THREAD id and `act()` posts it to `/api/queue/[id]`, which resolves against `triagedesk_drafts`. Measured by driving the page on 2026-09-19: the press answered `404 {"error":"draft not found"}` and the console showed the operator that sentence. No browser task is possible here. Defect 1. |
 unemploy4 recorded | open a protest draft (cd_drafts) | the table exists and no route anywhere in the product writes to it. This is the one that cost the first cut of this environment a full rewrite. |
 usingitup7 recorded | GET /rss.xml, GET /llms.txt, GET /brand/mark-email.png, GET /api/subscribe/unsubscribe, and every page (/, /about, /archive, /archive/all, /objects, /entry/[slug]) | they write nothing. `grep -rn ".insert(|.upsert(|.update(|.delete(|.rpc(" src/ scripts/ ops/` over the product tree returns exactly two lines, the upsert in /api/subscribe and the update in /api/subscribe/unsubscribe, and `grep -rn "use server" src/` returns nothing. Rule 3 says every reward reads database rows, so a reward on a pure read would have to grade a response body or an HTTP status, which is what a broken write looks like from the outside. |
 whyyourbraindoesthat7 recorded | compound-ops/letters/send-letter.py, the armed weekly letter lane | It writes publication_letter_sends only after a Resend send has succeeded, and it exits before any decision unless RESEND_API_KEY resolves, so --dry reaches no state and a real run mails strangers. Nothing may send real mail, so its one-letter-per-entry ledger and its unsubscribed=eq.false recipient filter are read here and never exercised. |
 wirecall6 recorded | POST /api/checkout | creates the order row and then calls Stripe. WireCall is on the SECOND live Stripe secret key on this machine, the one OutRip and MatchLine also carry. It runs here with a placeholder so its own unconfigured path runs; what that path leaves behind is in the defects instead. |