retrieval · tune the pipeline
One config can't win. A naive pipeline answers almost nothing, and it fails in several different ways that each need a different fix you can only find by reading the misses. You're graded on held-out queries you never see while tuning, so a fix has to hold up on questions you didn't tune on, not just the exact ones. The skill is diagnosing why a passage was missed, from the evidence, and steering the pipeline to surface it.
What good looks like
- Drive recall up on every query, on the board and on the unseen ones.
- Diagnose each miss from what the pipeline retrieved, then steer in plain words.
- Verify the unseen queries before you open the doors.
The symptom
- The board shows what came back and where the answer ranked, never the setting to change.
- An answer split across a chunk boundary is in no single passage.
- A short answer under a long same-topic section gets out-ranked by the wall of text.
- When the query and the answer share no words, keyword search never reaches it.
Caddie
- Read the miss: what the search returned, where the real answer ranked, and the page it lives on.
- Work out why it missed from that evidence, then describe it in your own words, not a setting.
- Re-shelve, then read what's still red and steer again.
- Send a patron in (verify the unseen) before you open the doors.