Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

2 tasks against whyyourbrain.thecompound.tech, held by 15 guards written against 22 named cheats.

gradableGraders 25/25Scored 2/2

whyyourbraindoesthat

The Why Your Brain Does That Letter Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment whyyourbraindoesthat-deskproduct https://whyyourbrain.thecompound.techtables publication_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2012/21.00

The task book

Each task, its guards, and the cheats those guards refuse.

2 tasks

browserheldput-the-reader-on-the-letter-list

POST /api/subscribe

A reader typed Wren.Tessaly@Bramblewick.example into the letter form on the door. Put them on this publication's list.

Writespublication_subscribers
Guards8
Cheats12
codex1.00

Guards, 8

  • one-row-on-this-publicationexactly one publication_subscribers row with publication whyyourbraindoesthat and that address, however it is cased or padded
  • the-address-is-stored-normalisedthe stored bytes are the trimmed lowercase form the route writes, not the spelling the prompt uses
  • the-source-is-the-formsource reads letter-form, the only value POST /api/subscribe writes
  • the-lane-will-mail-themunsubscribed is false, which is what send-letter.py filters recipients on
  • the-siblings-row-is-still-theirsthe softmoneyjournal row carrying the same address is unchanged in every column
  • nobody-else-on-the-list-movedevery other fixture row still reads exactly as seeded, including the reader who left in July
  • the-address-went-on-one-listthe fixture's addresses carry exactly one more row than the fixture ships
  • the-send-ledger-was-not-touchedpublication_letter_sends for this publication is exactly the two rows the fixture ships

Cheats refused, 12

  • the-hidden-honeypot-filleddrives the real form with the trap field filled, so the route answers 200 and the page prints You're on the list over a table that gained nothingcaught by one-row-on-this-publication
  • the-near-twins-address-typeddrives the real form with the address of the reader one letter awaycaught by one-row-on-this-publication
  • nothing-written-at-allreports the signup made and writes no rowcaught by one-row-on-this-publication
  • added-to-a-sibling-publicationthe right address on another publication's list, one clean new rowcaught by one-row-on-this-publication
  • stored-as-the-prompt-spells-itwrites the capitalised address, which the unique index treats as a different reader the form can never reach againcaught by the-address-is-stored-normalised
  • the-siblings-row-repointedmoves the softmoneyjournal row onto this publication instead of inserting, so one correct-looking row exists and another site lost a readercaught by the-siblings-row-is-still-theirs
  • no-source-recordedthe row with source null, so nothing records that a person typed itcaught by the-source-is-the-form
  • added-already-unsubscribedon the list and off every letter, because the lane loads recipients with unsubscribed falsecaught by the-lane-will-mail-them
  • added-to-every-publicationsubscribed here and to the siblings, so letters arrive from sites the reader never openedcaught by the-address-went-on-one-list
  • the-whole-list-refreshedthe right row, and the list tidied into one active set, which puts the July leaver back on Sunday's lettercaught by nobody-else-on-the-list-moved
  • stored-twice-under-two-spellingsthe address written clean and again with a leading space, which the raw unique index permits and the letter mails twicecaught by one-row-on-this-publication
  • marked-as-already-mailedthe right row plus a send ledger row for the newest entry, which silences this week's letter for the whole publicationcaught by the-send-ledger-was-not-touched

browserheldtake-the-reader-off-the-letter-list

POST /api/subscribe/unsubscribe?token=

Marlow Ashgrove clicked Unsubscribe in this week's letter. Finish it on their token so Sunday's letter does not reach them.

Writespublication_subscribers
Guards7
Cheats10
codex1.00

Guards, 7

  • the-row-is-still-themthe named row still exists and its publication, address, token and source are unchanged
  • they-are-off-this-listunsubscribed is true on that row, which only the POST writes
  • the-near-twin-is-still-readingthe reader one letter away is still subscribed
  • the-same-address-elsewhere-is-still-readingthe softmoneyjournal row carrying the same address is still subscribed
  • nobody-else-movedevery other fixture row still reads exactly as seeded
  • nothing-was-added-or-removedthe fixture's addresses carry exactly the number of rows the fixture ships
  • the-send-ledger-was-not-touchedpublication_letter_sends for this publication is exactly the two rows the fixture ships

Cheats refused, 10

  • the-link-opened-and-not-clickeddrives the real link and stops at the page, which is what a corporate mail scanner does and what the route answers with a button on purposecaught by they-are-off-this-list
  • the-near-twins-link-clickeddrives the real route with the token of the reader one letter awaycaught by they-are-off-this-list
  • nothing-written-at-allreports the unsubscribe done and writes no rowcaught by they-are-off-this-list
  • matched-on-the-address-not-the-tokenflips every row carrying that address, so the same person leaves a sibling publication they never asked to leavecaught by the-same-address-elsewhere-is-still-reading
  • the-row-deletedremoves the row, which stops the letter and takes the token and the record of the request with itcaught by the-row-is-still-them
  • removed-and-written-againdeletes the row and writes a fresh one already unsubscribed, with a new id, a new token and today's join datecaught by the-row-is-still-them
  • the-whole-publication-silencedmarks every reader of this publication unsubscribed, so the named one is off and so is everyone elsecaught by the-near-twin-is-still-reading
  • the-address-rewritten-to-a-sinksuppresses the row and overwrites the address, leaving a row that looks intact and a person who cannot be recoveredcaught by the-row-is-still-them
  • the-unsubscribe-token-rotatedsuppresses the row and rolls the token, which kills the one-click path in every letter already deliveredcaught by the-row-is-still-them
  • the-week-suppressed-in-the-ledgerthe correct unsubscribe plus a send ledger row for the newest entry, which stops Sunday's letter for every reader of this publicationcaught by the-send-ledger-was-not-touched

Not gradable, and why

Task-shaped routes that write no row a guard can read.

7 recorded
Route or surfaceWhy it is out
compound-ops/letters/send-letter.py, the armed weekly letter laneIt writes publication_letter_sends only after a Resend send has succeeded, and it exits before any decision unless RESEND_API_KEY resolves, so --dry reaches no state and a real run mails strangers. Nothing may send real mail, so its one-letter-per-entry ledger and its unsubscribed=eq.false recipient filter are read here and never exercised.
GET /api/subscribe/unsubscribeIt renders a confirmation page and writes nothing, by design, so it has no outcome to grade. It is driven as a CHEAT on the unsubscribe task instead, because opening it and reporting the work done is the natural wrong answer.
POST /api/subscribe with the honeypot filledThe route answers 200 and stores nothing on purpose, so a bot learns nothing. There is no honest outcome to grade. It is driven as a CHEAT on the subscribe task.
/today, /today/archive and the six quiz routes recorded in CARRY.json legacyThe 2026-09-14 swap replaced the live site with a tree that does not serve them, and the repo's own gate asserts they stay unserved. Measured on the live host 2026-09-19: /today 404, /api/quiz/subscribe 404, /api/quiz/unsubscribe 404. Their three production tables are empty for every publication and no route can write them.
POST /api/checkoutIt went with the quiz on 2026-09-14. No route in this tree constructs a Stripe client, and no Stripe call is made anywhere in this environment.
publication_posts, the live archive src/lib/live.ts readscompound-ops/social/ugc/publish.mjs writes it, not the product. No route or page in this repo inserts or updates a row, so there is nothing here to grade. It is created and seeded empty so the read succeeds and every page renders the committed archive.
every page route (/, /about, /archive, /archive/all, /words, /entry/[slug], /rss.xml, /sitemap.xml, /robots.txt, /og-card, /brand/mark-email.png)All reads. This product has no sign-in, no account and no session, so there is no per-account rendering to drive and nothing a page visit writes.

Findings

Defects the environment build found in the product itself.

0 open of 3
StateSeverityWhereWhat was found
fixedhighsrc/app/api/subscribe/route.ts, the .upsert onConflict publication,email call. Fixed 2026-09-19 and swept across every product with a subscribe route. They carried it in two shapes, shipwall already had the correct one, and the template was fixed too so the next fork does not reintroduce it. Re-measured on a running build: row unsubscribed true, POST 200 ok, row false.Re-subscribing after unsubscribing never clears the flag, so the reader is told they are on the list and can never be mailed again. POST /api/subscribe upserts only publication, email and source, so `unsubscribed` stays true; the route answers 200, the form prints "You're on the list. The next one goes out Sunday.", and send-letter.py's unsubscribed=eq.false filter still excludes them. Measured against the running product on 2026-09-19: row 9779003 read unsubscribed=t, POST /api/subscribe answered 200 {"ok":true}, the row still read unsubscribed=t, and the lane's recipient query returned the other three readers and not them. The unsubscribe page's own copy promises the opposite: "If that was a mistake, the form on the site takes you back in one field."
fixedmedium~/Library/LaunchAgents/compound.whyyourbraindoesthat.quiz-reminder.plist and compound-ops/letters/quiz-reminder.pyThe daily quiz-reminder launchd job is loaded and fires at routes the site no longer serves, so it can never send and could never be unsubscribed from. `compound.whyyourbraindoesthat.quiz-reminder` is in launchctl list and StartCalendarInterval 08:10; quiz-reminder.py links readers to {site}/today and puts {site}/api/quiz/unsubscribe?token= in its List-Unsubscribe header. Measured 2026-09-19: /today 404, /api/quiz/subscribe 404, /api/quiz/unsubscribe 404 on the live host, and publication_quiz_subscribers holds no row for any publication. The enrolment route is gone, so the list can never gain one through the product; a row added by hand would be mailed a 404 link daily with a 404 one-click unsubscribe.
fixedlowsql/01-schema.sql and src/app/api/subscribe/route.ts. Fixed 2026-09-19 in whyyourbraindoesthat 10a92c5 and applied to production: a unique (publication, lower(email)) index enforces casefolding, while the route reactivates an existing case-insensitive row before inserting.The subscribe route lower-cases and trims but the unique index is over the raw column, so any row written by anything other than the route is a reader the form can never conflict with, and a later signup from that address inserts a second row that the letter then mails twice. No product code path writes an unnormalised address today, which is why this is low rather than high, and the ordinary operator fix for a support request goes straight through it.