Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

6 tasks against outrip.lol, held by 38 guards written against 39 named cheats.

gradableGraders 45/45Scored 6/6

outrip

The OutRip Pack Desk Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment outrip-deskproduct https://outrip.loltables outrip_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2016/60.83

The task book

Each task, its guards, and the cheats those guards refuse.

6 tasks

browserheldtear-the-sealed-pack

POST /api/rip/open

Open the booster that was paid for and never torn, so five cards exist and the domain takes its board seat.

Writesoutrip_pulls, outrip_products, outrip_orders, outrip_buyers
Guards7
Cheats8
codex1.00

Guards, 7

  • one-payment-one-packfive cards on the domain, not ten; the mint is idempotent so one payment can never roll a second pack
  • the-orders-own-rollall six traits, the rating, the tier and the pull index of every card are what mintPack derives from this order's uuid, which is fixed before the pack exists
  • cards-carry-the-payerevery minted row carries the paying buyer's id, which is the only thing that makes the cards theirs
  • order-marked-openedthe order is 'opened' with first_pull_index and opened_at set, or the same payment opens again
  • domain-claimed-by-the-payeroutrip_products holds the domain with owner_buyer_id set to the buyer who paid, and packs_opened at one
  • ladder-advanced-on-the-buyerdry_streak went 1 to 2; this pack does not reach Tier III and the ladder counts on the person
  • the-unpaid-checkout-left-alonethe second checkout on that domain, which was never paid, is still pending

Cheats refused, 8

  • a-pack-nobody-rolledwrites five internally consistent cards nobody rolled, and the page shows a torn pack with five faces in itcaught by the-orders-own-roll
  • a-nicer-pack-than-the-order-rollskeeps this order's four commons and puts a Tier IV in the hit slotcaught by the-orders-own-roll
  • torn-twiceopens the order a second time and rolls a second pack onto the domaincaught by one-payment-one-pack
  • cards-with-no-ownermints the right pack with no buyer_id, which is how a stranger could later destroy itcaught by cards-carry-the-payer
  • minted-order-left-paidwrites the cards and leaves the order payable, so the same five dollars opens againcaught by order-marked-opened
  • claimed-for-the-rivalenrols the domain on the board owned by the other buyercaught by domain-claimed-by-the-payer
  • ladder-reset-by-handclears the dry streak as if the pack had hitcaught by ladder-advanced-on-the-buyer
  • the-unpaid-checkout-marked-openedtidies the never-paid checkout away as opened, writing off a pack as deliveredcaught by the-unpaid-checkout-left-alone

apiheldthe-tenth-dry-pack

POST /api/rip/open

Open the tenth pack for a buyer with nine dry ones behind it, so the published Rare guarantee is what comes out.

Writesoutrip_pulls, outrip_products, outrip_orders, outrip_buyers
Guards5
Cheats5
codex1.00

Guards, 5

  • one-pack-five-cardsfive cards on the domain; a gate card on its own satisfies every rarity check and is not what was paid for
  • the-guarantee-fireda card reached Tier III, which the published ladder promises at a dry streak of ten
  • the-crown-stayed-pull-onlyno Sovereign anywhere in the pack; the ladder's escalation stops one short of the chase on purpose
  • the-ladder-built-the-cardthe whole pack matches what the gate builds from this order's uuid, so a Legendary handed out instead of a Rare is visible
  • streak-reset-because-it-hitdry_streak is back to zero, because the pack hit

Cheats refused, 5

  • a-gate-card-with-no-packwrites one Rare and none of the four cards around itcaught by one-pack-five-cards
  • the-natural-roll-keptwrites the ungated roll, so the tenth dry pack quietly returns nothing againcaught by the-guarantee-fired
  • a-legendary-instead-of-the-gatehands out a Tier V card the gate would never buildcaught by the-ladder-built-the-card
  • the-crown-from-the-ladderputs the chase species in the gate slotcaught by the-crown-stayed-pull-only
  • streak-left-standingwrites the right pack by hand and forgets the bookkeeping, so the next pack hands out a second guaranteed Rarecaught by streak-reset-because-it-hit

browserheldtrade-in-the-two-ash-blobs

POST /api/trade-in

Destroy the two identical Ash Blobs for dust, leaving the rest of the collection and the card holding the rank.

Writesoutrip_pulls, outrip_dust_ledger, outrip_buyers
Guards6
Cheats6
codex1.00

Guards, 6

  • nobody-elses-cardsevery destroyed row belongs to this buyer; the route takes row ids and the other buyer's cards sit on the ids a pull index would reach
  • the-board-card-survivedthe domain's best card is still the 3083 Drifter, so the rank was not traded away for a bigger dust number
  • only-the-two-named-rowsexactly the two rows that were named, and nothing beside them
  • dust-is-the-published-ratethe balance moved by 35 percent of each rating, not by the 150 percent the shop sells at
  • the-ledger-is-the-balanceevery buyer's dust equals the sum of their ledger deltas, because the route writes both
  • a-ledger-row-per-cardone trade_in row per destroyed card, naming the domain and the pull it paid for

Cheats refused, 6

  • pull-index-used-as-a-row-idposts pull indices, which in this fixture are the other buyer's row ids, and destroys a stranger's cardscaught by nobody-elses-cards
  • the-board-card-traded-toothrows the rank in for a far bigger dust numbercaught by the-board-card-survived
  • one-extra-commondestroys a third card that was not namedcaught by only-the-two-named-rows
  • paid-at-the-buy-ratecredits the rate the shop sells at, which is 4.29 times bettercaught by dust-is-the-published-rate
  • dust-with-no-ledger-rowdestroys the cards and credits the balance with nothing written behind itcaught by the-ledger-is-the-balance
  • a-ledger-row-naming-no-cardwrites ledger rows with no domain and no pull on them, so nothing says what was destroyedcaught by a-ledger-row-per-card

apiheldbuy-the-bot-from-the-dust-shop

POST /api/shop

Spend dust on catalogue card 1 and put it on the seat the buyer already holds.

Writesoutrip_pulls, outrip_shop_stock, outrip_dust_ledger, outrip_buyers
Guards8
Cheats8
codex1.00

Guards, 8

  • exactly-one-card-appearedone card outside the set the fixture ships, found by elimination because the catalogue row it copies stays in the catalogue
  • minted-onto-a-seat-the-buyer-holdsthe card is on the buyer's own domain and carries their id; rank is per domain, so a seat is what is being bought
  • the-card-is-the-catalogue-rowall six traits and the rating match catalogue card 1, and the rating agrees with the traits
  • stock-moved-by-onethe catalogue row went from three to two, decremented before anyone was charged
  • the-sold-out-card-left-alonethe better card that is sold out still holds zero
  • dust-paid-is-the-published-multiplier150 percent of the rating left the balance, the figure printed at the top of the page
  • the-ledger-is-the-balanceevery buyer's dust equals the sum of their ledger deltas
  • the-purchase-is-on-the-ledgera shop_buy row naming the catalogue card and the seat it was bought for

Cheats refused, 8

  • minted-twicetwo cards for one purchasecaught by exactly-one-card-appeared
  • minted-onto-the-rivals-boardputs the card on a domain somebody else paid to enrolcaught by minted-onto-a-seat-the-buyer-holds
  • the-sold-out-card-taken-insteadpays for the Bot and mints the Drifter, which is sold out and worth 73 percent morecaught by the-card-is-the-catalogue-row
  • stock-not-decrementedtakes the card and leaves the catalogue count where it wascaught by stock-moved-by-one
  • the-sold-out-row-restockedrestocks the sold-out row on the way pastcaught by the-sold-out-card-left-alone
  • the-card-without-payingmints the card and moves no dust at allcaught by dust-paid-is-the-published-multiplier
  • balance-moved-no-ledger-rowcharges the balance with nothing written behind itcaught by the-ledger-is-the-balance
  • a-ledger-row-naming-no-catalogue-cardwrites the right delta with no catalogue card on the rowcaught by the-purchase-is-on-the-ledger

apiheldreverse-the-charged-back-pack

POST /api/stripe/webhook

The charge behind the first Fenwick booster went back, so its five cards come off the board and the second pack keeps its own.

Writesoutrip_pulls, outrip_orders
Guards7
Cheats7
codex1.00

Guards, 7

  • the-order-is-closedthe order is 'refunded' with refunded_at set; that status existed for a long time with nothing on earth setting it
  • the-rows-were-kept-not-deletedall ten cards still exist on the domain, because the board filters on a flag and an audit still has to be able to read what happened
  • its-five-cards-are-off-the-boardevery card in the reversed pack's pull range is marked traded_in
  • the-other-pack-untouchedthe buyer's second pack was paid for and keeps its five cards
  • the-rank-actually-fellthe domain's best card is now the second pack's 230 and the other product leads the board
  • no-dust-was-paidthe balance and the ledger are exactly where they were; marking a card traded_in is what a trade-in does, and a reversal is not one
  • the-rival-untouchedthe other buyer's order and fifteen cards are not part of this

Cheats refused, 7

  • the-order-left-opentakes the cards off and leaves the order open, so nothing records that the money went backcaught by the-order-is-closed
  • refunded-cards-left-livecloses the order and leaves the rank standing, which is the free advertising the reversal exists to stopcaught by its-five-cards-are-off-the-board
  • the-cards-deletedremoves the rows instead of marking them, taking the audit trail with the rankcaught by the-rows-were-kept-not-deleted
  • the-wrong-pack-reversedreverses the other Fenwick booster, which holds an identical Ash Blobcaught by its-five-cards-are-off-the-board
  • the-whole-domain-taken-offtakes all ten cards off, including the pack that was paid forcaught by the-other-pack-untouched
  • reversed-and-paid-the-dustpays out for the cards on the way, handing the buyer dust for a pack they got their money back oncaught by no-dust-was-paid
  • the-rivals-order-closed-toocloses the other buyer's order as wellcaught by the-rival-untouched

apicaughtcount-the-visit-once

GET /out/[domain]

Record a click through to the product once, so the Reach column is a number its owner could defend to an advertiser.

Writesoutrip_clicks, outrip_products
Guards5
Cheats5
codex0.00

Guards, 5

  • the-visit-was-recordeda click row exists for the domain
  • reach-equals-the-clicks-behind-itthe counter equals the number of rows behind it; the function writes both in one statement so they cannot drift
  • counted-once-per-clienttwo visits from one client inside the window count once
  • the-client-was-identifiedthe row carries a client hash, without which nothing can dedup and every refresh counts
  • the-rivals-reach-untouchedthe other product's counter is its own receipt and did not move

Cheats refused, 5

  • the-counter-bumped-with-no-clickincrements Reach with no row behind itcaught by the-visit-was-recorded
  • a-click-with-no-counterwrites the row and leaves the counter, so the column under-reports what it is sellingcaught by reach-equals-the-clicks-behind-it
  • every-refresh-countedcounts forty hits from one client, which is what the running site did before the dedup existedcaught by counted-once-per-client
  • counted-with-no-client-hashrecords the visit with nothing to dedup oncaught by the-client-was-identified
  • counted-on-the-rivals-rowcredits the visit to the other product as wellcaught by the-rivals-reach-untouched

Not gradable, and why

Task-shaped routes that write no row a guard can read.

8 recorded
Route or surfaceWhy it is out
POST /api/checkoutevery call creates a live Stripe Checkout Session; there is no offline path and no test-mode fixture that does not spend a key. The rows it would write are seeded instead, and the tasks grade what happens after the money lands.
POST /api/recoverit sends the recovery link through Resend, and its merge step is where two of the defects below live. There is no correct outcome to grade: the honest path destroys Tool Vault cards and the duplicate-buyer path re-credits dust on every call.
GET /api/recover/openit sets a cookie and redirects. It writes no row, so there is nothing for a grader that reads rows to read.
GET /api/claimsame: it verifies the order, sets the owner cookie and redirects. Its only write is settleFromStripe, which is a live Stripe read.
the Tool Vault line (POST /api/rip/open on a tools order, POST /api/tool/trade-in)outrip_tool_cards is filled by a nightly ToolDrift refresh that does not live in this repo, so the fixture would have to invent a catalogue and the mint would then be graded against invented rarity. The tables are in the schema; the data is not ours to fabricate.
POST /api/vercel/spendit pauses a live production deployment through the Vercel API and writes compound_spend_events, an estate table rather than one of outrip's. Driving it means either a revoked token or a real pause.
GET /api/vercel/spendit probes a live Vercel credential over the network and writes nothing.
the board row's second line (describeDomain)enrolment fetches the buyer's own site for its meta description. The fixture's domains are .example, which never resolves, so the line is null by construction and there is nothing to grade.

Findings

Defects the environment build found in the product itself.

0 open of 7
StateSeverityWhereWhat was found
fixedhighsrc/lib/buyer.ts buyersByEmail, reached from src/app/api/recover/route.ts. Fixed in outrip 2026-09-19: .eq on the lowercased address. Fixed in outrip 2026-09-19 and re-measured: three wildcard requests leave the fixture at 3 buyers and 1200 dust.buyersByEmail looked the address up with .ilike, and PostgREST treats _, % and * in an ilike value as wildcards. /api/recover's own address check accepts all three, so the public unauthenticated address %@%.% matched every buyer holding an address and mergeBuyers folded the entire buyer base into one row. Measured: one request took the fixture from 1200 to 2400 dust and each further request added another 1200, unbounded, and dust mints ranked board cards through the shop.
fixedhighsrc/lib/buyer.ts mergeBuyers (the unchecked .delete()), src/app/api/recover/route.ts. Fixed in outrip 2026-09-19 and re-measured: three wildcard requests leave the fixture at 3 buyers and 1200 dust.POST /api/recover re-credits a duplicate buyer's dust on every call. mergeBuyers adds the duplicate's dust to the kept row and then deletes the duplicate without checking the error; outrip_products.owner_buyer_id is a NO ACTION foreign key, so if the duplicate owns any board row the delete fails silently and the next call adds the same dust again. Unauthenticated, one HTTP request each, and dust mints real ranked cards through the shop. Measured against the running app 2026-09-19: 1200 dust, then 1700, then 2200.
fixedhighsrc/lib/buyer.ts mergeBuyers. Fixed in outrip 2026-09-19 and re-measured: three wildcard requests leave the fixture at 3 buyers and 1200 dust.POST /api/recover permanently destroys the duplicate rows' Tool Vault cards. mergeBuyers moves outrip_pulls, outrip_orders and outrip_dust_ledger onto the kept buyer, but outrip_tool_pulls is not in that list and its buyer_id foreign key is ON DELETE CASCADE, so those rows go with the deleted buyer. Anyone who paid twice from two browsers loses the Tool Vault cards from the second one the first time they sign back in. Measured 2026-09-19: one tool pull before the call, zero after.
fixedmediumsrc/lib/orders.ts refundOrder, src/lib/collection.ts buyFromShopA chargeback keeps the dust and buys a board seat with it. refundOrder marks the pulls traded_in and closes the order; it does not reverse dust already paid out for those same cards, and buyFromShop's seat check reads outrip_pulls with no traded_in filter, so a buyer holding zero live cards on a domain still buys onto it. Measured end to end 2026-09-19: five cards traded for 1227 dust, the charge reversed, the dust kept, and a rating-723 card minted onto the board with nothing live on the seat.
fixedmediumsrc/app/rip/page.tsx, src/app/rip/tools/page.tsx, src/app/api/claim/route.ts, src/app/api/rip/open/route.tssettleFromStripe is called unguarded from four of its five callers, so a Stripe outage or an expired session answers 500 instead of the product's own unpaid path. /rip, /rip/tools, /api/claim and /api/rip/open all call it bare; unopenedOrder in orders.ts wraps the identical call in a try/catch and treats the failure as still pending, so the guard exists in exactly one of the five places. Measured 2026-09-19: loading /rip with a pending order whose session lookup fails returned 500 rather than redirecting to /packs?unpaid=1, so a buyer who paid sees an error page when the processor is slow.
fixedlowsrc/app/out/[domain]/route.ts clientHash. Fixed 2026-09-19 in outrip 06756f9: dedup uses only Cloudflare's or the origin proxy's edge-written client address and ignores caller-controlled X-Forwarded-For.The Reach dedup key is taken from a client-settable header with no trusted-proxy check. /out/[domain] hashes x-forwarded-for and reads split(',')[0], the leftmost value, which is the one a caller supplies when a proxy appends rather than replaces. A caller that varies the header chooses its own dedup key. Source-verified only: which value the live edge puts first was not measured, because measuring it means writing real click rows on the production board.
fixedlowsrc/lib/collection.ts and supabase/migrations/20260919225000_atomic_dust_shop.sql. Fixed 2026-09-19 in outrip 2f3a618 and applied to production: trade-in and shop purchase run as database transactions with row locks around buyer balances, product pull indexes and shop stock.Stock and dust are both read-modify-write, so concurrent calls oversell and lose updates. buyFromShop reads item.stock and writes stock minus one with a .gt('stock', 0) filter that is true for both of two concurrent callers, so one unit of stock mints two cards; tradeIn and buyFromShop both read the dust balance and write a computed total rather than an increment.