Building a checkout with an AI and calling the score a benchmark is easy. Whether that score means anything is the hard part. If a half-decent model would pass the test no matter what you typed, the test is grading the model, not you — and most prompt benchmarks never stop to check which one they’re doing. So before a hidden test goes anywhere near the leaderboard, it has to clear one bar.
A hidden test earns its place only if a naive spec fails it and an expertspec passes it. Naive means you named the task but not the quirky convention; expert means you pinned the convention down. And I don’t take my own word for it — I run both specs through the real build-and-grade pipeline N times and watch what actually happens.
Each hole keeps two honest specs on file: a naive one and an expert one. I push both through the exact pipeline a real player hits — same Kimi build on a live Daytona sandbox, same Playwright checks — and log how often each spec clears each hidden test. What I’m hunting for is a gap. If the expert clears a test and the naive spec doesn’t, the test is doing its job. If the naive spec clears it too, the test is telegraphed: the model handles it on autopilot, so it’s measuring the model, not you. Those get reworked or cut. The harness has no private build of its own and no private grader of its own. It runs through production, because that’s the only number that isn’t flattering.
There are language models inside a round — it would be dishonest to pretend otherwise — so I keep them on a leash you can inspect. The points that rank you come from deterministic checks: Playwright driving the real build, or a harness diffing your module against a reference oracle. No model opinion can add or remove a hidden-test point. A model shows up in exactly four places: a gate that rejects gibberish before I spend a build on it, a craft judge that reads your spec, a visual judge that looks at screenshots of what shipped, and a post-round diagnosisthat tells you whether your gap was prompting or domain knowledge. The two judges can only touch the small style band — fifteen points, clearly labeled. The diagnosis can touch nothing at all: the score is computed and locked before it runs, and its output is words, not points.
And when a judge is unavailable — a key missing, a provider down — its band drops out and the remaining weights renormalize. The scorecard says so, in plain text. What I will not do is fake a style score, silently swap one model for another, or let an LLM’s vibe leak into the fifty-five points that hidden tests own. If a number on your card can’t be reproduced by a machine re-running the same checks, it isn’t on your card.
Agent builds aren’t deterministic. The same spec can clear a test on one run and miss it on the next, so “it discriminated once” means almost nothing — I’ve watched a test that looked airtight quietly flip on a fresh sample. One run is a coin toss. Eight naive fails against eight expert passes is a result. I won’t ship a hole on fewer than five runs, and I won’t put a number on this page under ten.
Take the rate-limiter hole. It hides one finicky rule: the client key has to be canonicalized — trimmed and lowercased — soUSER:Alice anduser:alicedraw from one budget instead of two. If you’ve shipped a rate limiter you write that without thinking. If you haven’t, it never crosses your mind — and an attacker just rotates the capitalization to print free requests.
| hidden test | naive | expert | verdict |
|---|---|---|---|
| Spelling variants share one bucket | 0/5 | 5/5 | ✓ discriminates |
| Differential fuzz | 0/5 | 5/5 | ✓ discriminates |
Average across the runs: naive 35, expert 100. Every hidden test splits the two specs cleanly. That’s a hole I’m happy to put on the leaderboard.
When a hidden test turns out telegraphed — a naive spec passes it too — the report says so, and I fix it or kill it. And when even a sharp expert spec can’t reliably pass a test, that’s usually not the player’s fault; it’s the builder model hitting its ceiling. I mark those holes frontier and keep them off the ranked leaderboard. I’d rather catch my own weak holes than have someone catch them for me on launch day.