evaluation · build the judge
This is the real AI-engineering job: the oracle doesn't exist until you build it. A generic 'is it good?' rubric scores near random, because a capable judge can't know your refund window is 30 days or that you don't offer phone support. The skill is translating a messy spec into a discriminating eval, and you're graded by how well your eval agrees with ground truth, including cases you never saw while building it.
What good looks like
- Author an eval that catches the defective answers and passes the clean ones.
- Ground it in the policy; a generic rubric can't see the factual violations.
- Verify it on held-out cases before you finish, where overfitting shows up.
The symptom
- A capable judge catches the obvious defects on its own, so those don't separate you.
- The subtle defects are policy violations only your rubric can encode.
- Flag everything and precision collapses; flag nothing and recall is zero.
- Your eval is scored on cases you can't see while you build it.
Caddie
- Read the policy, then read each answer against it.
- Write a rubric that encodes the specific facts that matter, not generic quality.
- Submit, read your precision and recall, and tighten what you missed.
- Verify on the held-out cases before you finish.