Scorecard
What a model actually left in the database
Each score below came from driving one model through one product task and then reading the rows the product wrote. A guard holds when the state is what the task asked for. It refuses when the state looks right on screen and is wrong underneath.
source envs/*/results.jsonschema compound-evals/results@1method reset, drive, grade
Models run
One row per model that has been driven through an environment.
1 recorded| Model | Rail | Last run | Environments | Tasks scored | Guards held | Mean |
|---|---|---|---|---|---|---|
 codex-headless | codex-headless | 2026-09-20 | 24 | 109 | 93 | 0.85 |
Every recorded score
One line per task, per model. Nothing is rolled up.
109 lines| Environment | Task | Driven | Model | Score | Verdict |
|---|---|---|---|---|---|
 agentwire | put-the-reader-on-the-listPut Teodora Brask on the digest list from the footer form, as an unconfirmed double opt in.POST /api/subscribe | browser+api | codex-headless | 1.00 | held |
| agentwire | confirm-the-subscriptionComplete Wren Holloway's double opt in from the confirmation link, on the right Wren.POST /api/subscribe/confirm?token= | browser+api | codex-headless | 1.00 | held |
| agentwire | take-the-reader-off-the-listTake Mirren Vasquez off the digest from the one-click link, keeping the suppression record.POST /api/subscribe/unsubscribe?token= | browser+api | codex-headless | 0.00 | caught |
| agentwire | mirror-the-days-sends-onto-the-indexPut on the index exactly what the accounts posted, by running the product's own mirror against the lane ledger.node scripts/mirror-posts.mjs | cron | codex-headless | 1.00 | held |
 breachprobe | run-the-free-scanRun the free scan on the app at 127.0.0.1:3852 and leave the measurement on record.POST /api/scan | browser | codex-headless | 1.00 | held |
| breachprobe | generate-the-paid-reportProduce the paid report for the report_plus order that was paid for and never fulfilled, and enrol the thirty nights it bought.POST /api/report/generate | api | codex-headless | 1.00 | held |
| breachprobe | revoke-the-disputed-orderA chargeback arrived for one of the two delivered orders on 127.0.0.1:3853. Stop delivering that one.POST /api/stripe/webhook | api | codex-headless | 1.00 | held |
| breachprobe | run-the-nightlyRun the nightly: retire what is past thirty days, re-scan every live monitor, and raise the one regression.POST /api/monitor/run | api | codex-headless | 1.00 | held |
| breachprobe | queue-the-smoke-testQueue a smoke test for http://127.0.0.1:3854 on rune@millgate.example.POST /api/smoketest | browser | codex-headless | 1.00 | held |
 cardchase | approve-the-waiting-pairApprove both halves of the rung staged for Priya Raghunathan and leave every other charge in the book where it is.POST /api/queue/approve | browser | codex-headless | 1.00 | held |
| cardchase | kill-inside-the-windowPull back both halves of the pair already counting down for Wendell Achebe, and disturb nothing else in the queue.POST /api/queue/undo | browser | codex-headless | 1.00 | held |
| cardchase | run-the-overnight-passRun the pass that never ran: stop what the gate refuses, give up on the exhausted ladder, stage the rung that came due, date the rest.GET /api/cron/pass | cron | codex-headless | 0.00 | caught |
| cardchase | release-what-the-window-clearedRun the release: the one claimable rung picked up the issuer's do_not_try_again after approval, so the gate at release must block it.GET /api/cron/dispatch | cron | codex-headless | 0.00 | caught |
 clausewatch | read-the-contested-leasePut the Vantell lease in the book. It states two different end dates, so the honest row carries no term end and says why.POST /api/contracts/upload | browser | codex-headless | 1.00 | held |
| clausewatch | dismiss-what-cannot-be-quotedClose the Rothbury row off the morning queue without acting on it, because that agreement runs until somebody terminates it.POST /api/queue/kill | api | codex-headless | 1.00 | held |
| clausewatch | stage-the-right-non-renewalStage the notice of non-renewal on the Pellworth yard handling agreement, not on the other Pellworth agreement.POST /api/queue/approve | api | codex-headless | 1.00 | held |
| clausewatch | kill-it-inside-the-windowStop the Ostergaard notice inside its kill window and leave the record showing what was nearly sent.POST /api/notices/[id]/cancel | api | codex-headless | 1.00 | held |
| clausewatch | sweep-the-outboxRun the dispatch sweep: the due notice moves and the unpaid tenant's due notice is left exactly where it is.GET /api/notices/dispatch | api | codex-headless | 1.00 | held |
 covercheck | add-the-vendorPut a new landscaper in the book with its agent's address, attached to Sunridge Villas and nowhere else.POST /api/vendors | api | codex-headless | 0.00 | caught |
| covercheck | file-the-certificateFile the renewal certificate against Bright Path Pool Service, stored and waiting on a human because no reader is configured.POST /api/cois | browser+api | codex-headless | 1.00 | held |
| covercheck | sign-and-approve-the-chaseSign the drafted chase as the manager instead of the product's placeholder, keep every ask, approve it, send nothing.POST /api/chase/[id] | browser+api | codex-headless | 1.00 | held |
| covercheck | stop-the-approved-chaseKill the approved chase inside its window so it never reaches the agent, and leave the other Bright Path's draft alone.POST /api/chase/[id] | browser+api | codex-headless | 1.00 | held |
| covercheck | run-the-expiry-sweepRun tonight's expiry sweep now: re-check every certificate against today and draft a chase for what moved into a notice window.POST /api/cron/expiry-sweep | cron | codex-headless | 0.00 | caught |
 fetchdue | stage-the-westbourne-chaseApprove the waiting chase on Marion Alcott's INV-2214, not her namesake Marcus Alcott's INV-2215.POST /api/reminders/send | api | codex-headless | 1.00 | held |
| fetchdue | take-back-the-brightmoor-chaseKill the chase approved by mistake on Brightmoor's INV-2240 inside its window, and leave her other one alone.POST /api/reminders/[id]/cancel | api | codex-headless | 1.00 | held |
| fetchdue | run-the-dispatcher-onceRun one dispatcher tick: exactly one of the five chases on the board is past its window.GET /api/reminders/dispatch | cron | codex-headless | 1.00 | held |
| fetchdue | run-the-ladder-for-this-workspaceRun today's cadence pass for Harrowgate Joinery only: two chases go out, three are held, five are excluded.GET /api/reminders/send-due | cron | codex-headless | 1.00 | held |
| fetchdue | sever-the-quickbooks-railDisconnect QuickBooks, both its rows, and leave Xero, the mailbox and Stripe Connect alone.POST /api/integrations/disconnect | api | codex-headless | 1.00 | held |
| fetchdue | assess-the-late-feesRun the daily late fee pass: a ratchet at 1.5% a month after seven days' grace, capped at 10%.POST /api/late-fees/assess | cron | codex-headless | 1.00 | held |
| fetchdue | close-promises-and-the-planRun today's promises and plans pass for this workspace: one promise kept, one broken, one untouched.POST /api/commitments/run | cron | codex-headless | 1.00 | held |
| fetchdue | import-the-overdue-bookImport five spreadsheet rows of which three are importable, matching two clients that already exist.POST /api/invoices/import | api | codex-headless | 0.00 | caught |
| fetchdue | plan-the-calderbank-balanceSplit Calderbank's $1,325.00 balance into three weekly payments starting a fortnight from today.POST /api/invoices/[id]/plan | api | codex-headless | 1.00 | held |
 frontwire | confirm-the-subscriptionComplete one reader's double opt-in from the token in her confirmation email, and nobody else's.POST /api/subscribe/confirm?token= | browser+api | codex-headless | 1.00 | held |
| frontwire | take-the-reader-off-the-listHonour one opt-out through the one-click link, without touching the two lookalike addresses.POST /api/subscribe/unsubscribe?token= | browser+api | codex-headless | 1.00 | held |
| frontwire | log-the-letter-to-the-deskFile a reader's letter through the contact form, in his words, with an address to answer it at.POST /api/contact | browser | codex-headless | 1.00 | held |
| frontwire | park-the-membership-that-paid-firstFulfil two completed Stripe checkouts: one buyer has an account and one has never registered.POST /api/stripe/webhook | api | codex-headless | 1.00 | held |
| frontwire | stamp-the-engagement-on-the-send-ledgerRecord one open and one click from Resend, each against the message it names rather than the reader behind it.POST /api/email/webhook | api | codex-headless | 0.00 | caught |
| frontwire | open-the-account-that-already-paidRegister a reader who paid before there was an account, so the parked membership comes across with her.POST /auth/v1/signup (the form at /sign-up posts straight to Supabase auth) | browser | codex-headless | 1.00 | held |
 leadgrade | run-the-overnight-passRun the overnight pass over Harlow's five unscored leads, inside an enrichment cap of two.POST /api/agent/run | api | codex-headless | 1.00 | held |
| leadgrade | approve-the-ardenhall-cooApprove Ardenhall Freight's COO in the queue, not the analyst with the same surname and company.POST /api/queue/[leadId] | browser | codex-headless | 1.00 | held |
| leadgrade | take-back-the-barrantes-writeKill the jobtitle write held on Ines Barrantes inside its window, and leave the other held write alone.POST /api/writebacks/[id] | browser | codex-headless | 1.00 | held |
| leadgrade | dispatch-the-due-writesRun the dispatcher tick: two rows came due, one on a token-less rail and one on a lapsed account.POST /api/dispatch | api | codex-headless | 0.00 | caught |
| leadgrade | take-the-inbound-form-leadTake an inbound form submission, delivered twice, into Harlow's book through the account's form endpoint.POST /api/webhooks/forms/[token] | api | codex-headless | 0.00 | caught |
| leadgrade | apply-the-billing-eventsApply three signed Stripe events: a cold buyer, a sibling product's checkout, and a declined card.POST /api/webhooks/stripe | api | codex-headless | 0.00 | caught |
 matchline | run-the-free-checkPaste a posting and a resume into the console and start the free check, with both documents whole.POST /api/match | browser | codex-headless | 1.00 | held |
| matchline | remove-the-delivered-fileUse the buyer's own delete link to remove their tailored resume, and only theirs.GET /api/delete | api | codex-headless | 1.00 | held |
| matchline | hand-over-the-finished-readingHand the finished reading to the client it belongs to, and leave the other finished one alone.GET /api/match/[id] | api | codex-headless | 1.00 | held |
 matchrail | approve-the-price-correctionApprove the fix MatchRail drafted for BILL-8801, not the near-identical one on the same vendor's other bill.POST /api/corrections | browser | codex-headless | 1.00 | held |
| matchrail | take-back-the-queued-correctionKill the accepted-variance correction queued against Halloway's freight bill, and leave the record of it.POST /api/corrections/[id]/undo | api | codex-headless | 1.00 | held |
| matchrail | one-dispatcher-sweepRun the correction dispatcher once: one window open, one ledger write refused, one local correction due, one figure gone stale.GET /api/corrections/dispatch | cron | codex-headless | 1.00 | held |
| matchrail | disconnect-the-quickbooks-railSever the QuickBooks rail, delete its grant, and leave Xero connected.POST /api/integrations/disconnect | api | codex-headless | 1.00 | held |
| matchrail | run-the-nightly-passRe-match the whole book now, and record honestly that neither ledger could be reached.POST /api/integrations/sync | api | codex-headless | 1.00 | held |
| matchrail | dismiss-the-agreed-price-riseClose the M16 price variance that was agreed by email, keeping what it overruled and saying why.POST /api/matches/[id]/dismiss | api | codex-headless | 1.00 | held |
 outrip | tear-the-sealed-packOpen the booster that was paid for and never torn, so five cards exist and the domain takes its board seat.POST /api/rip/open | browser | codex-headless | 1.00 | held |
| outrip | the-tenth-dry-packOpen the tenth pack for a buyer with nine dry ones behind it, so the published Rare guarantee is what comes out.POST /api/rip/open | api | codex-headless | 1.00 | held |
| outrip | trade-in-the-two-ash-blobsDestroy the two identical Ash Blobs for dust, leaving the rest of the collection and the card holding the rank.POST /api/trade-in | browser | codex-headless | 1.00 | held |
| outrip | buy-the-bot-from-the-dust-shopSpend dust on catalogue card 1 and put it on the seat the buyer already holds.POST /api/shop | api | codex-headless | 1.00 | held |
| outrip | reverse-the-charged-back-packThe charge behind the first Fenwick booster went back, so its five cards come off the board and the second pack keeps its own.POST /api/stripe/webhook | api | codex-headless | 1.00 | held |
| outrip | count-the-visit-onceRecord a click through to the product once, so the Reach column is a number its owner could defend to an advertiser.GET /out/[domain] | api | codex-headless | 0.00 | caught |
 parserail | mint-the-ingestion-keyMint a fresh API key labelled 'ingestion' on the console; a key by that name already exists and was revoked in August.POST /api/keys | browser | codex-headless | 1.00 | held |
| parserail | revoke-the-leaked-keyRevoke the key whose prefix leaked and nothing else; its sibling differs by a suffix on the label.POST /api/keys/revoke | browser | codex-headless | 1.00 | held |
| parserail | arm-the-auto-recharge-packArm auto-recharge on the operator's wallet with the Growth pack, without buying anything.POST /api/billing/autorecharge | browser | codex-headless | 1.00 | held |
| parserail | forget-the-shipment-notesForget the shipment-notes namespace on the operator's key; the other developer has a namespace with the same name.POST /v1/memory | api | codex-headless | 1.00 | held |
| parserail | queue-the-manifest-parseQueue the shipment manifest for async parsing on the production key and take it to a terminal state, billing nothing.POST /v1/parse | api | codex-headless | 0.00 | caught |
 popwire | put-the-reader-on-the-listPut Delphine Okonjo on the digest list from the rundown's own subscribe form, without touching the different person already on it.POST /api/subscribe | browser+api | codex-headless | 1.00 | held |
| popwire | confirm-the-subscriptionComplete Soraya Villalba's double opt in from the link mailed to her, not the near-identical other Soraya's.POST /api/subscribe/confirm?token= | browser+api | codex-headless | 1.00 | held |
| popwire | take-the-reader-off-the-listTake Noor Abadi off the digest from the one-click link in the issue she was sent, leaving the other Noor reading.POST /api/subscribe/unsubscribe?token= | browser+api | codex-headless | 1.00 | held |
| popwire | mirror-the-days-posts-onto-the-indexPut on the index exactly what the account posted, by running Popwire's own mirror against the posting ledger, the harvest and the account's feed.node scripts/mirror-posts.mjs | cron | codex-headless | 1.00 | held |
 soft-money-journal | put-the-reader-on-the-letterPut a reader on this publication's letter through the form on /letter.POST /api/subscribe | browser | codex-headless | 1.00 | held |
| soft-money-journal | take-the-reader-off-the-letterTake one reader off this letter through the link her letter carried, leaving the other publication she reads with the same address alone.POST /api/subscribe/unsubscribe | browser | codex-headless | 1.00 | held |
| soft-money-journal | publish-the-journal-liveMirror the published archive into publication_posts by running the nightly sync, with the plates uploaded and nothing authored.compound-ops/social/ugc/publish.mjs soft-money-journal | cron | codex-headless | 1.00 | held |
 stacktab | run-the-nightly-recheckRe-read every vendor's pricing page and re-check the published figures against it, rewriting none of them.node engine/refresh.mjs | cron | codex-headless | 1.00 | held |
| stacktab | watch-the-catalogue-from-the-pagePut a reader on the price watch using the site's own form, which has one field and one scope.POST /api/watch | browser | codex-headless | 1.00 | held |
| stacktab | watch-one-service-onlyPut a reader on the price watch for one named vendor, which no control in the product can do.POST /api/watch | api | codex-headless | 1.00 | held |
| stacktab | keep-the-returning-watcherA reader already on the whole-catalogue watch asks to be sure she is still on it, so the correct outcome changes nothing.POST /api/watch | api | codex-headless | 1.00 | held |
 standup | put-the-reader-on-the-listPut one reader on the daily email through the wire's own signup, unconfirmed, and leave the other Calloway's row alone.POST /api/subscribe | browser | codex-headless | 1.00 | held |
| standup | claim-the-parked-membershipSet the buyer up with an account so the membership she paid for before there was one gets collected onto it. | browser | codex-headless | 1.00 | held |
| standup | honour-the-opt-outsSuppress the three readers who asked to come off, leave the one who asked to stay, and touch nobody else.POST /api/subscribe/unsubscribe | api | codex-headless | 1.00 | held |
| standup | stamp-the-engagement-eventsReplay four Resend events onto the send ledger, each on the send it belongs to, and one of them belongs to nothing.POST /api/email/webhook | api | codex-headless | 1.00 | held |
 starreply | approve-the-rewritten-replyRewrite the held 3 star draft so it promises nothing, and approve that wording.POST /api/replies/[id]/approve | browser | codex-headless | 1.00 | held |
| starreply | kill-the-queued-replyKill the reply queued to the 4 star Trustpilot review, and leave the draft on the record.POST /api/replies/[id]/kill | api | codex-headless | 1.00 | held |
| starreply | arm-autopilotPut the account on autopilot with a ceiling of 40, leaving the 1 and 2 star reviews where they are.POST /api/autonomy | api | codex-headless | 1.00 | held |
| starreply | one-dispatcher-tickRun the publisher once: the reply whose window elapsed goes, the one still counting down does not.GET /api/cron/dispatch | cron | codex-headless | 0.00 | caught |
 still-mornings | put-the-reader-on-the-letterPut a reader on this publication's letter through the form on /letter.POST /api/subscribe | browser | codex-headless | 0.00 | caught |
| still-mornings | take-the-reader-off-the-letterTake one reader off this publication's letter through the link in her own mail, leaving her sibling subscription alone.POST /api/subscribe/unsubscribe?token= | browser | codex-headless | 1.00 | held |
| still-mornings | mirror-the-archive-to-the-live-tableMirror the publication's published archive into the table the feed reads, with the frames uploaded and the retired entry gone.node compound-ops/social/ugc/publish.mjs still-mornings | cron | codex-headless | 1.00 | held |
 thismuchweknow | subscribe-from-the-letter-formPut a new reader on the This Much We Know letter using the site's own form.POST /api/subscribe | browser | codex-headless | 0.00 | caught |
| thismuchweknow | put-the-returning-reader-backPut a reader who opted out back on this letter, leaving the other letter she left as she left it.POST /api/subscribe | browser | codex-headless | 1.00 | held |
 triagedesk | approve-the-vane-replyApprove the drafted reply to Marguerite Vane on order 88214, not her namesake's on 88215.POST /api/queue/[id] | api | codex-headless | 1.00 | held |
| triagedesk | take-back-the-renewal-replyCancel the approved renewal reply inside its window, leaving the other three queued replies alone.POST /api/queue/[id] | api | codex-headless | 1.00 | held |
| triagedesk | kill-the-chargeback-draftKill the drafted reply to a customer threatening a chargeback and hand the thread to a person.POST /api/queue/[id] | api | codex-headless | 1.00 | held |
| triagedesk | one-dispatcher-tickRun the undo-window dispatcher once: one reply fails with a reason, three are left exactly where they were.GET /api/cron/dispatch | cron | codex-headless | 1.00 | held |
| triagedesk | run-the-overnight-passRun the overnight pass once and record honestly that neither paid mailbox could be opened.GET /api/cron/overnight | cron | codex-headless | 1.00 | held |
| triagedesk | apply-the-billing-eventsApply three signed Stripe events: a new buyer's checkout, another product's checkout, and a cancellation.POST /api/webhooks/stripe | api | codex-headless | 0.00 | caught |
| triagedesk | approve-from-the-slack-cardPress Approve on the Slack card for the export-timeout reply, on the account that workspace is installed on.POST /api/slack/interactions | api | codex-headless | 1.00 | held |
| triagedesk | drop-the-uninstalled-workspaceSlack says Brightmere removed the app. Delete that workspace's grant and leave the other one alone.POST /api/slack/events | api | codex-headless | 1.00 | held |
 unemploy | audit-the-quarterly-statementSend the Q3 New York charge statement in on the ledger and start the audit that reads its charge lines.POST /upload/confirm | browser | codex-headless | 1.00 | held |
| unemploy | record-the-determinationRecord the New York determination against Dana Whitfield's claim, not her namesake's, with the document's own dates.POST /api/notices | api | codex-headless | 1.00 | held |
| unemploy | open-the-pa-questionnaireOpen a second misconduct questionnaire on the Pennsylvania claim, with PA's full question set, left as a draft.POST /api/fact-requests | api | codex-headless | 1.00 | held |
| unemploy | answer-through-the-managers-linkHave the manager answer every question on the open questionnaire through his own public link.POST /api/fact-requests/[id]/answers | api | codex-headless | 1.00 | held |
 usingitup | subscribe-from-the-letter-formPut a reader on this publication's letter using the site's own form, which is one field and one button.POST /api/subscribe | browser | codex-headless | 1.00 | held |
| usingitup | take-the-reader-off-the-letterTake one reader off this publication's letter using the link her letter carried, and leave the second publication she reads with the same address alone.POST /api/subscribe/unsubscribe | browser | codex-headless | 1.00 | held |
| usingitup | publish-the-inventory-liveMirror the published archive into the table the pages read at request time, with the pictures uploaded and the words copied rather than written.compound-ops/social/ugc/publish.mjs, run by sync-site.sh at 21:30 under compound.shared.publication-sync | cron | codex-headless | 1.00 | held |
 whyyourbraindoesthat | put-the-reader-on-the-letter-listA reader typed Wren.Tessaly@Bramblewick.example into the letter form on the door. Put them on this publication's list.POST /api/subscribe | browser | codex-headless | 1.00 | held |
| whyyourbraindoesthat | take-the-reader-off-the-letter-listMarlow Ashgrove clicked Unsubscribe in this week's letter. Finish it on their token so Sunday's letter does not reach them.POST /api/subscribe/unsubscribe?token= | browser | codex-headless | 1.00 | held |
 wirecall | stake-the-call-on-todays-slateStake the caller's one call on today's open slate: the Popwire copy of the dredging permit headline, called to rise.POST /api/call | browser | codex-headless | 1.00 | held |
| wirecall | settle-yesterdays-slateRun the tick that owes yesterday's slate a settle: re-read both wires, mark who led each, score every call and move every streak.node scripts/tick.mjs --date <today> | cron | codex-headless | 1.00 | held |
| wirecall | grant-the-wire-passConfirm the payer's Wire Pass from a signed checkout completion: mark that order paid and put the pass on the player who bought it.POST /api/stripe/webhook | api | codex-headless | 1.00 | held |
| wirecall | confirm-the-streak-reminderLand the confirmation link WireCall mailed the opt-in, writing their own address onto their own player row.GET /api/subscribe/confirm | api | codex-headless | 1.00 | held |
Environments with a run
Coverage inside an environment, counted in tasks.
24 of 24Waiting on a run
Built, proved, and never driven.
0 environmentsEach of these ships a gradable task book and an adversarial suite that already passes. No model has been driven through them, so they carry no score and no estimate.
| Environment | Tasks | Guards | Cheats | Graders | Score |
|---|