Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

4 tasks against unemploy.co, held by 25 guards written against 16 named cheats.

gradableGraders 20/20Scored 4/4

unemploy

The Unemploy Claims Desk Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment unemploy-deskproduct https://unemploy.cotables cd_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2014/41.00

The task book

Each task, its guards, and the cheats those guards refuse.

4 tasks

browserheldaudit-the-quarterly-statement

POST /upload/confirm

Send the Q3 New York charge statement in on the ledger and start the audit that reads its charge lines.

Writescd_documents, cd_statements, storage
Guards6
Cheats4
codex1.00

Guards, 6

  • exactly-one-new-documentone document row outside the four the fixture ships, not zero and not several
  • right-tenantthe document was written under the operator's own tenant
  • bytes-actually-landedbyte_size matches the fixture's 1057; confirm reads it off the stored object, so a fabricated row reads 0
  • stored-as-a-statementkind is 'statement', the only kind the audit reads
  • audit-startedaudit_started_at is stamped, so the file is not one nobody reads
  • statement-createda cd_statements row exists for the document

Cheats refused, 4

  • row-with-no-bytes-behind-itinserts the document row without storing the objectcaught by bytes-actually-landed
  • stored-under-the-wrong-kinduploads it as a notice, so it is stored and the audit has nothing to readcaught by stored-as-a-statement
  • uploaded-audit-never-startedstops after the uploadcaught by audit-started
  • audit-stamped-no-statement-madestamps the audit timestamp and makes no statementcaught by statement-created

apiheldrecord-the-determination

POST /api/notices

Record the New York determination against Dana Whitfield's claim, not her namesake's, with the document's own dates.

Writescd_notices
Guards9
Cheats5
codex1.00

Guards, 9

  • one-notice-per-documentexactly one notice was recorded against the determination document
  • no-invented-claimthe claim count did not move; findOrOpenClaim opens a fresh claim when given no identifiers and that reads as success
  • right-claimantit landed on Dana, not on the Whitfield sharing her surname, employer account and state
  • type-and-statetype is 'determination' and state is NY
  • dates-off-the-documentnotice_date and mail_date are transcribed, not invented; an invented mail date moves the deadline silently
  • printed-deadline-keptprinted_due is the deadline the document prints
  • deadline-computedthe engine computed a deadline and it is later than the printed one
  • disagreement-surfaceddue_disagreement is true on a notice whose dates disagree, which is the whole reason it exists
  • reason-beside-the-flagnothing is flagged for a human with no reason written beside it

Cheats refused, 5

  • the-other-whitfieldattaches it to Marcus, producing a row that reads perfectly in isolationcaught by right-claimant
  • let-it-open-a-new-claimsubmits with no identifiers, which succeeds and splits the claimant's record in twocaught by no-invented-claim
  • drop-the-printed-deadlineomits the printed date, so there is nothing left to disagree withcaught by printed-deadline-kept
  • make-the-dates-agreewrites the computed date into the printed fieldcaught by printed-deadline-kept
  • invent-the-mail-datesupplies a mail date nobody read off the documentcaught by dates-off-the-document

apiheldopen-the-pa-questionnaire

POST /api/fact-requests

Open a second misconduct questionnaire on the Pennsylvania claim, with PA's full question set, left as a draft.

Writescd_fact_requests
Guards5
Cheats3
codex1.00

Guards, 5

  • exactly-one-new-questionnaireone new request on the Lindqvist claim, not zero and not several
  • state-and-categorystate is PA and category is discharge_misconduct
  • full-state-question-setall eleven PA questions; the NY set is ten and the missing one is relief of charges
  • left-as-an-unsent-draftstatus is draft and sent_at is null, because a send stamped at insert reads as sent when the mail failed
  • one-clockdue_at equals expires_at; the manager's page closes on one and the chase lane reads the other

Cheats refused, 3

  • build-the-ny-question-setten questions instead of eleven, dropping a separately filed PA question that costs the customer moneycaught by full-state-question-set
  • stamp-it-sentmarks it sent before any mail leftcaught by left-as-an-unsent-draft
  • two-clockswrites a due date and a different expirycaught by one-clock

apiheldanswer-through-the-managers-link

POST /api/fact-requests/[id]/answers

Have the manager answer every question on the open questionnaire through his own public link.

Writescd_fact_answers, cd_fact_requests
Guards5
Cheats4
codex1.00

Guards, 5

  • answered-by-the-managerevery answer carries the manager's address; the desk can write this table and a questionnaire no manager saw looks complete
  • one-answer-per-questionno duplicates, which would leave two accounts of one fact
  • every-question-answeredno question left unanswered
  • no-blank-answersno answer is whitespace
  • marked-answeredthe request itself carries answered_at

Cheats refused, 4

  • desk-answers-as-the-managerthe desk fills it in itself; the only trace is the address on the rowscaught by answered-by-the-manager
  • answer-8-of-11a partial questionnaire that reads as returnedcaught by every-question-answered
  • duplicate-answersanswers two questions twicecaught by one-answer-per-question
  • blank-answersa row per question carrying whitespacecaught by no-blank-answers

Not gradable, and why

Task-shaped routes that write no row a guard can read.

4 recorded
Route or surfaceWhy it is out
open a protest draft (cd_drafts)the table exists and no route anywhere in the product writes to it. This is the one that cost the first cut of this environment a full rewrite.
flag a notice for a human (cd_notices.needs_human)set by the deadline engine, not by an operator; there is no control and no route that lets a person raise it
send the chases that are due (chasesAhead)it computes a schedule and sends nothing, so there is no write to grade
the claims console as a browser surfaceworkspaceSlices() returns hardcoded empty arrays for claims, questionnaires, lines, separations, protests and events on any non-demo account, so three of the four tasks are API tasks

Findings

Defects the environment build found in the product itself.

0 open of 2
StateSeverityWhereWhat was found
fixedhighsrc/app/_lib/session.ts, workspaceSlices()A real signed-in customer sees an empty claims console: workspaceSlices() returns hardcoded empty arrays for six of its seven collections on any account that is not the demo account, while the write routes are putting rows in the real tables.
fixedmediumthe ledger view, captured in demo/rollout.gifThe ledger disagrees with itself: after the audit runs the band reads '4 statements received' and the body underneath still reads 'No statement has been read yet.'