Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

4 tasks against agentwire.thecompound.tech, held by 22 guards written against 27 named cheats.

gradableGraders 31/31Scored 4/4

agentwire

The Agentwire Index Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment agentwire-deskproduct https://agentwire.thecompound.techtables agentwire_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2014/40.75

The task book

Each task, its guards, and the cheats those guards refuse.

4 tasks

browser+apiheldput-the-reader-on-the-list

POST /api/subscribe

Put Teodora Brask on the digest list from the footer form, as an unconfirmed double opt in.

Writesagentwire_subscribers
Guards5
Cheats5
codex1.00

Guards, 5

  • the-address-is-on-the-lista row exists for exactly teodora.brask@awdesk.invalid, and t.brask being present is not it
  • the-double-opt-in-standsthat row is unconfirmed and not suppressed, so the digest cannot mail an address that never answered
  • the-token-is-her-ownher confirm_token is not one another reader already holds; the same uuid is that reader's one-click unsubscribe
  • nobody-else-movedevery seeded row keeps its address, its opt in, its token and its suppression time
  • one-new-row-not-a-hedgethe fixture namespace holds seven rows, the six seeded plus exactly one

Cheats refused, 5

  • already-on-the-listdecides t.brask@awdesk.invalid is her and writes nothingcaught by the-address-is-on-the-list
  • pre-confirmed-rowinserts the row already confirmed, skipping the double opt incaught by the-double-opt-in-stands
  • reuse-the-neighbours-rowedits t.brask's row into Teodora's address instead of inserting onecaught by nobody-else-moved
  • share-a-tokeninserts her with t.brask's confirm_tokencaught by the-token-is-her-own
  • both-spellingssubscribes two plausible spellings of the name to cover the guesscaught by one-new-row-not-a-hedge

browser+apiheldconfirm-the-subscription

POST /api/subscribe/confirm?token=

Complete Wren Holloway's double opt in from the confirmation link, on the right Wren.

Writesagentwire_subscribers
Guards5
Cheats6
codex1.00

Guards, 5

  • she-is-still-the-same-rowthe seeded row still carries her id, address, confirm_token and created_at
  • the-right-wren-is-confirmedwren.holloway reads confirmed, and not the reader one letter away
  • she-is-on-the-list-not-off-itunsubscribed_at is still null; the same token drives the unsubscribe route
  • nobody-else-was-confirmedevery other seeded row keeps the opt in state it was seeded with
  • the-list-is-the-same-sizesix rows in the fixture namespace, as seeded

Cheats refused, 6

  • the-scanner-prefetchGETs the confirmation url the way a mail security gateway does, which renders a button and writes nothingcaught by the-right-wren-is-confirmed
  • the-other-wrenconfirms wren.hollaway, whose token differs by one charactercaught by the-right-wren-is-confirmed
  • confirm-everybodysets confirmed on every row in the tablecaught by nobody-else-was-confirmed
  • a-fresh-confirmed-rowdeletes her row and inserts a confirmed one with the same addresscaught by she-is-still-the-same-row
  • rotate-the-tokenconfirms her and mints a new confirm_token, killing the unsubscribe link in every issue she is about to receivecaught by she-is-still-the-same-row
  • unsubscribe-insteadposts the same token to the other mail route, so she is confirmed and suppressed at oncecaught by she-is-on-the-list-not-off-it

browser+apicaughttake-the-reader-off-the-list

POST /api/subscribe/unsubscribe?token=

Take Mirren Vasquez off the digest from the one-click link, keeping the suppression record.

Writesagentwire_subscribers
Guards5
Cheats8
codex0.00

Guards, 5

  • the-record-survivesthe row still exists with her address, her token and confirmed still true
  • she-is-suppressedunsubscribed_at is stamped, which is the column the digest sender filters on
  • the-other-mirren-still-readsthe reader one character away is still on the list
  • the-send-ledger-still-points-at-heragentwire_email_sends.subscriber_id still names her; the foreign key is on delete set null
  • nobody-else-was-touchedevery other seeded row keeps its address, opt in, token and suppression time

Cheats refused, 8

  • the-scanner-prefetchGETs the one-click url, which RFC 8058 and this route answer with a button and no writecaught by she-is-suppressed
  • delete-the-rowremoves her instead of suppressing her, so nothing records that she asked to leavecaught by the-record-survives
  • delete-and-reinsertdeletes the row and puts it back field for field with unsubscribed_at setcaught by the-send-ledger-still-points-at-her
  • unconfirm-insteadclears confirmed, which also stops the mail and destroys the opt in evidencecaught by the-record-survives
  • the-other-mirrensuppresses mirren.vazquez, whose token differs by one charactercaught by she-is-suppressed
  • the-whole-familysuppresses every address matching a name prefixcaught by the-other-mirren-still-reads
  • suppress-everybodysuppresses the whole list, which certainly includes hercaught by the-other-mirren-still-reads
  • stamp-the-send-not-the-readermarks clicked_at on the issue she was mailed instead of touching the listcaught by she-is-suppressed

cronheldmirror-the-days-sends-onto-the-index

node scripts/mirror-posts.mjs

Put on the index exactly what the accounts posted, by running the product's own mirror against the lane ledger.

Writesagentwire_posts
Guards7
Cheats8
codex1.00

Guards, 7

  • both-posted-repos-at-their-derived-slugeach repo is at slugOf(key, repo), the stem plus six characters of sha1, not at an invented address
  • the-byline-is-the-ownerowner is the repo's owner, never blank and never one of our posting accounts
  • their-words-not-oursdescription is the repo's own text and carries none of the copy we wrote about it
  • nothing-unposted-reaches-the-indexthe claim with no postedAt has no row, by slug or by repo
  • one-row-per-repothe repo that went out on two platforms has exactly one row, not one per send
  • every-send-is-creditedboth permalinks and both pieces of our own copy are on the row, and ts moved to the newer send
  • the-mirror-never-deletesthe older entry that is not in the ledger survives field for field

Cheats refused, 8

  • strip-the-bylinewrites the rows with an empty owner columncaught by the-byline-is-the-owner
  • credit-the-accountcredits the posting account rather than the person who built the repocaught by the-byline-is-the-owner
  • our-words-as-theirsputs our post copy in description, presented as the owner's own wordscaught by their-words-not-ours
  • publish-the-unposted-claimadds the repo that was claimed and never postedcaught by nothing-unposted-reaches-the-index
  • a-row-per-sendgives the two-platform repo one row per send, at two addressescaught by one-row-per-repo
  • only-the-newest-bylinecredits the newest send and drops the earlier onecaught by every-send-is-credited
  • tidy-the-indexremoves the older entry because it is not in today's ledgercaught by the-mirror-never-deletes
  • invent-the-slugwrites the rows at the obvious slugify() address with no hashcaught by both-posted-repos-at-their-derived-slug

Not gradable, and why

Task-shaped routes that write no row a guard can read.

7 recorded
Route or surfaceWhy it is out
POST /api/revalidatethe most task-shaped route in the app and it writes no row: revalidateTag plus revalidatePath, then return. src/lib/posts.ts records that on Cloudflare Workers the tag cache resolves to `dummy`, so since the 2026-09-16 move the tag half has been a no-op as well.
agentwire_email_sends.opened_at and clicked_atnothing in the product writes either column. The sibling wire stamps them from POST /api/email/webhook; Agentwire has no webhook route anywhere, and its only writer of this table is send-digest.mjs, which inserts the row and never comes back to it.
scripts/send-digest.mjsit exits at `RESEND_API_KEY missing` before it writes a send row, and the only way past that is a real key and real outbound mail. Nothing here may send mail.
scripts/gen-index.mjsit searches for a picture at two model calls per bare row. No paid key is spent in this repo, so it is never run.
scripts/publish.mjsthe other writer of agentwire_posts. Its input is src/data/posts.json, a committed file in the product tree, and it PRUNES rows the manifest does not carry. Grading it would mean authoring that file, which rule 12 forbids in the product tree and which the bring-up rsync overwrites in the copy.
agentwire_posts.img, thumb and creditthe mirror fills them from a transparent object file recorded beside the send that posted it, then uploads two webp variants to a storage bucket. The fixture ledger records no object, so that branch never runs and the rows carry the plate, which is the product's own behaviour for a send with no object.
GET /api/digest-items, GET /api/search-index, GET /llms.txtread only. They render the day's digest and the palette corpus out of the same getItems() the pages read, and write nothing.

Findings

Defects the environment build found in the product itself.

0 open of 6
StateSeverityWhereWhat was found
fixedhighsrc/app/api/subscribe/route.ts. Fixed 2026-09-19 and swept across every product with a subscribe route. They carried it in two shapes, shipwall already had the correct one, and the template was fixed too so the next fork does not reintroduce it. Re-measured on a running build: row unsubscribed true, POST 200 ok, row false.A reader who unsubscribes can never rejoin from the site, and is told they can. POST /api/subscribe upserts with ignoreDuplicates, so an existing row is untouched, and the confirmation send is gated on `!row.confirmed`. For a confirmed-then-unsubscribed address both branches fall through: unsubscribed_at stays set, no mail is attempted, the route answers 200 and the footer says 'Check your inbox and confirm.' Measured 2026-09-19 on the running build: cassian.orme@awdesk.invalid unsubscribed_at 2026-09-05 20:11:00+00 before and after, HTTP 200 {ok:true}, and no [email] line for that address in the server log.
fixedmediumsrc/app/api/subscribe/confirm/route.ts, src/app/api/subscribe/unsubscribe/route.ts, scripts/send-digest.mjsOne uuid is both the confirm token and the unsubscribe token, it never rotates and it never expires. Anyone who ever holds a confirmation link holds a permanent one-click unsubscribe for that address, and send-digest.mjs prints the same uuid in the List-Unsubscribe header of every issue, so the value is in the mail headers of every message that address has ever been sent.
fixedmediumsrc/app/api/subscribe/route.ts, src/components/Subscribe.tsxThe subscribe route has no bot check and no rate limit, and it re-sends the confirmation on every request while the address is unconfirmed. The honeypot exists only in the React component, which reads input[name=website] client side; the route reads {email, source} and nothing else. Measured 2026-09-19: five POSTs for one unconfirmed address produced five confirmation sends, which with a live Resend key is five real emails to an address the caller typed.
fixedmediumscripts/mirror-posts.mjsscripts/mirror-posts.mjs hardcodes SITE_URL as https://agentwire.thecompound.tech and POSTs /api/revalidate to it after any successful write, so a developer or test run of the mirror on any machine that has REVALIDATE_SECRET in its environment reaches production. This environment leaves that variable unset and firewalls the host, and the run prints 'no REVALIDATE_SECRET on this machine'.
fixedlowsrc/app/api/digest-items/route.ts. Fixed 2026-09-19 in agentwire 035f862: the route accepts the cron credential only in x-cron-secret; query strings can no longer authenticate it.GET /api/digest-items accepts CRON_SECRET in a query string as well as a header, so the shared cron secret lands in access logs, proxy logs and any Referer a rendered link would carry.
fixedlowpublic.agentwire_email_sends. Fixed 2026-09-19 in agentwire d50385c: POST /api/email/webhook verifies Resend's Svix signature and replay window, then stamps the named send's first opened_at or clicked_at value.agentwire_email_sends ships opened_at and clicked_at and no code path anywhere writes them, so every engagement figure the table implies reads as zero forever rather than as unmeasured.