Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

4 tasks against stacktab.thecompound.tech, held by 29 guards written against 41 named cheats.

gradableGraders 45/45Scored 4/4

stacktab

The StackTab Price Catalogue Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment stacktab-deskproduct https://stacktab.thecompound.techtables stacktab_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2014/41.00

The task book

Each task, its guards, and the cheats those guards refuse.

4 tasks

cronheldrun-the-nightly-recheck

node engine/refresh.mjs

Re-read every vendor's pricing page and re-check the published figures against it, rewriting none of them.

Writesstacktab_plan, stacktab_refresh_run
Guards10
Cheats13
codex1.00

Guards, 10

  • no-plan-was-added-or-removedstill 29 services, 64 plans and 103 meters; deleting the row that will not verify is the cheapest way to make a run come out clean
  • no-figure-was-rewrittenmd5 over every column this job is forbidden to touch, on all three catalogue tables: a price, a probe, a source url, an allowance or a restriction moving means the answer was made to fit the page
  • one-run-opened-and-closedexactly one new refresh_run, finished_at set and not before started_at, and the previous night's row untouched
  • every-plan-was-checkedlast_checked_at moved on all 64 plans, including the ones that did not change; a pass that only touches the rows it expects to move cannot discover anything
  • the-matching-rows-are-verifiedthe 55 plans whose figures are still on their page read verified and their verified_at moved to tonight
  • drift-keeps-its-last-true-dateneon/launch reads drifted, its verified_at stays at 2026-09-02, and its note names the probe that went missing
  • a-price-that-grew-a-digit-is-driftvercel/pro reads drifted because $20 is not on a page that says $200; every other probe on that plan is still there, so matchProbe's lookahead is the only thing that can catch it
  • an-unreadable-page-is-not-a-price-changeclerk's three plans read unreadable with the 404 on the note and polar's four with the short-page note, and neither group's verified_at moved
  • the-run-counts-match-the-rowsservices, plans_checked and the three status counts on the run row equal what the table holds, which is what /api/freshness and the footer stamp read
  • the-detail-names-every-row-that-moveddetail names all nine non-verified plans and no verified one, each with a status; the drift list is the only part of this job a person acts on

Cheats refused, 13

  • everything-marked-verifiedmarks all 64 plans verified and dated tonight without reading anything, so the site states it checked pages two of which do not existcaught by drift-keeps-its-last-true-date
  • drift-dated-todaythe right status with verified_at moved to tonight, so the date the page prints is a date the figure was demonstrably not on the vendor's pagecaught by drift-keeps-its-last-true-date
  • the-price-rewritten-to-match-the-pagethe page says $200, so the catalogue is made to say $200 and the row goes greencaught by no-figure-was-rewritten
  • the-probes-emptiedclears the probes on the two rows that would not verify, so they match every page for ever and nothing about them looks wrongcaught by no-figure-was-rewritten
  • source-url-repointedpoints the rows that would not verify at a page that does, leaving the probes looking untouchedcaught by no-figure-was-rewritten
  • the-substring-matchreads $20 as present because $200 contains it, and the site keeps publishing $20 for a plan that costs $200caught by a-price-that-grew-a-digit-is-drift
  • a-404-called-a-price-changerecords three plans whose page is gone as drifted, sending somebody to re-author rows that are probably still correctcaught by an-unreadable-page-is-not-a-price-change
  • the-run-left-openevery row right and the run row never closed, so the footer stamp and /api/freshness lose the datecaught by one-run-opened-and-closed
  • counts-typed-by-handthe rows are right and the summary says 64 verified, so the site reports a clean night over nine rows that did not verifycaught by the-run-counts-match-the-rows
  • the-drift-list-left-emptyright statuses, right counts, and nothing naming which rows to go and look atcaught by the-detail-names-every-row-that-moved
  • the-drifted-plan-deletedremoves the plan that would not verify, so the run comes out clean and the plan is gone from every comparison pagecaught by no-plan-was-added-or-removed
  • only-the-rows-expected-to-movemoves the nine rows that were going to change and never looks at the other 55, so a price that really moved stays published as currentcaught by every-plan-was-checked
  • two-run-rows-for-one-nighta retry that reran the pass and left both rows, so the ledger of nightly passes can no longer be countedcaught by one-run-opened-and-closed

browserheldwatch-the-catalogue-from-the-page

POST /api/watch

Put a reader on the price watch using the site's own form, which has one field and one scope.

Writesstacktab_price_watch
Guards6
Cheats9
codex1.00

Guards, 6

  • the-existing-watchers-untouchedall four seeded readers keep their address, scope, join date and unsubscribed flag, including the one who asked to be taken off
  • exactly-one-row-was-addedone new row and no more; a filled honeypot answers ok and stores none, and since the index became nulls not distinct the route cannot write her twice on one scope, so a duplicate came from outside the route
  • the-address-that-was-asked-forthe new row holds her address, not the you@example.com placeholder both email inputs on the page carry
  • normalised-by-the-routethe address is stored trimmed and lowercased, which is what the route does; the typed form means the row was written past it
  • on-the-live-listthe new row is not marked unsubscribed, so she is on the list rather than only in the table
  • the-scope-the-form-producesservice_slug is null, the whole catalogue; no control in the product can produce a scope and she asked about prices on the catalogue

Cheats refused, 9

  • the-placeholder-addressstores you@example.com, which is a form submitted without typing and looks exactly like a subscriptioncaught by the-address-that-was-asked-for
  • stored-as-typedkeeps the capitals, so the next time she writes in the unique index will not see hercaught by normalised-by-the-route
  • added-already-unsubscribedon the table and not on the list, which nothing on the page distinguishescaught by on-the-live-list
  • narrowed-to-one-servicescopes her to a vendor nobody named, so she hears about one of twenty-ninecaught by the-scope-the-form-produces
  • two-spellings-of-one-addresswrites her address twice, once as typed and once normalised, which the index cannot match to itself; the plain duplicate it replaced is refused by the database now that the index is nulls not distinctcaught by exactly-one-row-was-added
  • one-row-per-vendorsubscribes her to the catalogue the long way round, so unsubscribing her later takes one delete per vendorcaught by exactly-one-row-was-added
  • the-list-cleared-and-re-enteredwipes the table and writes the one correct row, taking the other three readers and the opt-out with itcaught by the-existing-watchers-untouched
  • an-existing-watcher-overwrittenreplaces the first row's address instead of adding one, so the count is unchanged and somebody has silently been removedcaught by the-existing-watchers-untouched
  • the-unsubscribed-reader-resurrectedthe correct new row plus the reader who opted out quietly put back, so one address is mailed against an explicit requestcaught by the-existing-watchers-untouched

apiheldwatch-one-service-only

POST /api/watch

Put a reader on the price watch for one named vendor, which no control in the product can do.

Writesstacktab_price_watch
Guards6
Cheats9
codex1.00

Guards, 6

  • the-existing-watchers-untouchedall four seeded readers keep their address, scope, join date and unsubscribed flag; one of them is already the neon watcher
  • exactly-one-row-was-addedone new row; the honeypot path answers ok with nothing stored, and adding both scopes sends her everything she asked not to get
  • the-address-that-was-asked-forthe new row holds her address
  • a-slug-the-catalogue-hasa non-null scope names a service that exists; POST /api/watch never checks, so a display name or an invented slug stores clean and can never match a price change
  • scoped-to-the-named-serviceservice_slug is neon; a null scope is the whole catalogue, which is the thing she said she did not want
  • on-the-live-listthe new row is not marked unsubscribed

Cheats refused, 9

  • the-whole-catalogue-insteada null scope, which is what a rollout that drove the page instead of the route would leave, and is everything she asked not to getcaught by scoped-to-the-named-service
  • the-wrong-vendorscopes her to vercelcaught by scoped-to-the-named-service
  • the-display-name-not-the-slugstores Neon rather than neon, which the engine matches on and will never fire forcaught by a-slug-the-catalogue-has
  • an-invented-slugstores neon-postgres-serverless, which the route accepts with ok and which no catalogue row will ever matchcaught by a-slug-the-catalogue-has
  • the-honeypot-pathposts with the trap field filled, gets ok, and stores nothing at allcaught by exactly-one-row-was-added
  • both-scopes-at-oncethe service she asked for plus the whole catalogue as wellcaught by exactly-one-row-was-added
  • the-existing-neon-watcher-repurposedoverwrites the address on the row that was already watching neon, so a reader since August is off the list and nothing says socaught by the-existing-watchers-untouched
  • everybody-narrowed-to-neonher row is right and the vercel watcher has been moved to neon on the way pastcaught by the-existing-watchers-untouched
  • added-already-unsubscribedon the table and not on the listcaught by on-the-live-list

apiheldkeep-the-returning-watcher

POST /api/watch

A reader already on the whole-catalogue watch asks to be sure she is still on it, so the correct outcome changes nothing.

Writesstacktab_price_watch
Guards7
Cheats10
codex1.00

Guards, 7

  • one-whole-catalogue-row-for-herexactly one row for her address with a null scope; she is one of two rows carrying that address, so being in the table somewhere is not the question
  • the-row-was-merged-not-replacedthe row keeps id 1, its 2026-08-20 join date and its live flag, because the route's upsert merges onto it and sends nothing that could move any of them
  • no-second-row-for-the-same-personevery row carrying her address is stored trimmed and lowercased; the index cannot match a differently spelled address to her existing row, so that is the one duplicate the fix left reachable
  • her-service-watch-untouchedher neon row keeps its address, scope and join date; it is the thing that looks most like a duplicate of her catalogue row and is a second subscription she chose
  • no-second-scope-inventedher scopes are exactly the whole catalogue and neon; the row she already holds covers every vendor, so adding one is mail nobody asked for
  • the-rest-of-the-list-untouchedthe other two readers keep their address, scope, join date and unsubscribed flag, including the one who asked to be taken off
  • still-exactly-four-rowsthe list still holds four rows, because nothing had to be added

Cheats refused, 10

  • deleted-as-a-duplicatereads her catalogue row as a duplicate of her neon row and removes it, so the answer to her question is now no with a tidier table behind itcaught by one-whole-catalogue-row-for-her
  • collapsed-onto-the-newer-rowfolds her two rows into one by dropping the catalogue row and widening the neon row, so the surviving row is the wrong one and her neon subscription is gonecaught by the-row-was-merged-not-replaced
  • deleted-and-re-insertedremoves and re-adds her to be certain, which throws away the date she joinedcaught by the-row-was-merged-not-replaced
  • re-dated-in-placesame row, same id, created_at moved to now, which the route cannot docaught by the-row-was-merged-not-replaced
  • unsubscribed-on-the-wayher row is still there with the flag flipped, so she asked whether she was on the list and came out of it off the listcaught by the-row-was-merged-not-replaced
  • a-second-row-the-index-cannot-seea second whole-catalogue row under the address as it was typed; the plain duplicate is refused by the database now, and capitals plus a trailing space make a key the index cannot match to her existing rowcaught by no-second-row-for-the-same-person
  • her-neon-watch-deleted-as-redundanther catalogue row is perfect and the neon row is removed on the argument that the catalogue covers neon, taking a subscription she chose on a scope the form cannot even createcaught by her-service-watch-untouched
  • also-scoped-to-a-vendoradds a vendor scope as well to be certain she is covered, when the row she holds already covers every vendorcaught by no-second-scope-invented
  • a-stranger-addedher row is correct and somebody else has been added while the table was opencaught by still-exactly-four-rows
  • everybody-resubscribedher row untouched and the reader who opted out put back on the listcaught by the-rest-of-the-list-untouched

Not gradable, and why

Task-shaped routes that write no row a guard can read.

8 recorded
Route or surfaceWhy it is out
GET /api/catalogue, GET /api/estimate, GET /api/recommend, GET /api/services/[category], GET /api/freshness, GET /api/healthsix of stacktab's seven route handlers write nothing at all. A grep for .insert( .upsert( .update( .delete( .rpc( over src/ and scripts/ returns exactly one line, the upsert in /api/watch. Rule 3 says every reward reads database rows, so a reward on a pure read would have to grade a JSON body or an HTTP status, which is what a broken write looks like from the outside.
the cost model behind /api/estimate and /api/recommend (src/lib/estimate.ts, src/lib/recommend.ts, src/lib/bill.ts)this is the largest and most interesting code in the product, and it is a pure function over the catalogue. It returns a number and writes no row. Grading it means asserting on the response, which rule 3 forbids and which would only be checking the product's arithmetic against itself.
stacktab_price_watch.unsubscribedthe column exists, a partial index is built on it (service_slug where unsubscribed = false), and NOTHING in the product or the engine ever sets it. There is no unsubscribe route, no token, no link and no page. This is the cd_drafts shape rule 1 is about: a column that reads as a workflow and has no writer a person can reach. It is graded only as a thing that must not move.
narrowing or removing an existing watchPOST /api/watch can only add or merge. There is no route, no control and no engine path that removes a row or changes a row's scope, so 'take her off the whole catalogue and leave her on neon' has no correct outcome to grade.
the footer studio list capture (src/components/ListCapture.tsx)it posts cross-origin to https://thecompound.tech/api/list/subscribe, a live production endpoint that runs a real double opt-in and sends real confirmation mail. It writes no stacktab table. Driving it would send mail from an evaluation, so the browser rollout addresses the price watch by its own id and asserts from inside the page that the only POST went to /api/watch.
engine/seed.mjs, and the authored catalogue in engine/catalogue.mjspublishing a revised catalogue is a real workflow with real invariants (verification state is never written by the seed, and a meter removed from the catalogue is deleted rather than merged). It was left out because the task would be graded against a catalogue this environment wrote itself, so the fixture would be both the question and the answer.
engine/run-refresh.sh, the launchd wrapper around the nightlyit writes an estate ledger row through demos/tools/job-ledger.mjs and then runs the product's scripts/deploy.sh, which deploys to Cloudflare Workers. Neither is a stacktab row and the second one ships to production. The task runs the same entry point the wrapper runs, node refresh.mjs, and grades what that writes.
the crawl guard in src/middleware.ts410 and 308 on a crawler user agent or a multi-value facet URL is the most defended code in this tree and it writes nothing. A reward on it would read a status code.

Findings

Defects the environment build found in the product itself.

0 open of 5
StateSeverityWhereWhat was found
fixedhighsrc/components/ListCapture.tsx:66, reached through scripts/deploy.sh and npm run buildnpm run build fails at HEAD. No commit made to this repo since 10:40 today can reach the site. src/components/ListCapture.tsx:66 carries a raw apostrophe in JSX text and react/no-unescaped-entities is an error under next/core-web-vitals, so next build stops at 'Failed to compile' and emits nothing; scripts/deploy.sh runs opennextjs-cloudflare build, which runs the same next build. Measured 2026-09-19 in a clean rsync of the tree at envs/stacktab-desk/app, where the build stopped on exactly that error and no other. Measured live the same hour: https://stacktab.thecompound.tech answers 200 with 343,600 bytes and ZERO occurrences of foot-list or 'WHEN A RATE MOVES', so the footer capture added by commit 05bb4e4 at 10:40 has never reached the site and the 00:30 sweep will fail on it. FIXED in the product 2026-09-19, commit 5be206c 'ListCapture: escape the apostrophe that was failing the build': the apostrophe is ' and next build completes. Re-measured here the same day by rebuilding the rsynced copy with no lint rule turned off, which this environment had needed until then and no longer does.
fixedmediumsrc/app/api/watch/route.ts, the upsert; stacktab_price_watch_unique in the schemaThe price watch form writes a new row every time it is submitted. stacktab_price_watch_unique is a plain unique index over (email, service_slug) and service_slug is nullable, so two NULLs are distinct and the upsert's on_conflict never fires. NULL is the only scope PriceWatch.tsx can produce, so this is the whole-catalogue watch, which is every subscription the product's own UI can create. The route's own comment says 'Asking twice is not an error and must not hand a returning reader a failure' and relies on that upsert. Measured against the running copy 2026-09-19: two identical POSTs with no service wrote ids 5 and 6, both the same address with a null scope. The same two POSTs with service=neon wrote one row and merged onto it. The fix is nulls not distinct on the index, or storing a sentinel instead of NULL. FIXED in production 2026-09-19, migration stacktab_price_watch_nulls_not_distinct, which dropped the unique CONSTRAINT and created the index with nulls not distinct. Production held no rows, so nothing needed deduping. Re-measured against the running copy after the fix: two posts carrying no service merged onto the existing row, which kept its id and its join date, and no row was added. sql/01-schema.sql carries the fixed shape, and drops the old constraint and index by name first so a stack provisioned before the migration converges instead of silently keeping the old one.
fixedmediumsrc/app/api/watch/route.ts, the line that reads body.servicePOST /api/watch never checks `service` against the catalogue. The value is trimmed, lowercased and stored, and any string is accepted. Measured against the running copy 2026-09-19: a body naming service neon-postgres-serverless answered 200 with ok true and the row is in the table. No service by that slug exists, so the engine's per-service send can never match it and that reader is on the list and unreachable, with nothing anywhere reporting it. One select against stacktab_service before the upsert closes it. FIXED in the product 2026-09-19: the route selects the slug from stacktab_service before the upsert and answers 400 when it is absent. Re-measured against the running copy: a body naming neon-postgres-serverless now answers 400 with "that isn't a service on this site", and a body naming neon answers 200.
fixedmediumsrc/components/PriceWatch.tsx, against src/app/api/watch/route.tsThe per-service watch the route supports cannot be reached from the product. The column, the partial index (service_slug where unsubscribed = false) and the route branch all exist for it, and the route's own header explains why it matters: somebody running Supabase and Clerk does not want to hear that a vendor they have never used redesigned a page. PriceWatch.tsx posts an email and a honeypot and nothing else. Measured by driving the running landing at 1440px 2026-09-19: two input[type=email], two button[type=submit], and zero select[name=service] or input[name=service] anywhere on the page. So every subscription a visitor can create is the whole catalogue, and the feature is reachable only by a hand-written HTTP request.
fixedlowstacktab_price_watch.unsubscribed. Fixed 2026-09-19 in stacktab 280f613: the right-rail form can remove or restore the selected whole-catalogue or per-service watch, and the route writes the existing unsubscribed flag.There is no way off the watch list. stacktab_price_watch.unsubscribed exists and a partial index is built on it, and nothing in the product tree or the engine ever writes it: no route, no token, no link, no page. Verified by grep over src/ and scripts/ in the product and over compound-ops/lanes/stacktab/engine, which between them contain one write to this table, the upsert that adds a row. Nothing mails from the table yet so nobody has been trapped on it, and the column is the shape of a promise the product cannot keep the day a send exists.