← Log
LABDraft2025-07-10 · updated 2025-08-15
Grounded retrieval: first lessons
Early findings from building a retrieval system that answers operational questions only from cited runbooks.
- Topics
- Applied AI
- Retrieval
- Runbooks
- Tags
- ai
- rag
- runbooks
- reliability
Chunking by heading is not enough
Runbooks are procedural. A chunk that ends mid-step is worse than no chunk, because the model will synthesise the missing step.
The best results so far come from chunking by procedure boundary and carrying the previous heading as metadata.
Refusal is a feature
An answer that cites the wrong source is more damaging than an honest 'I do not know'. The evaluation set now includes refusal cases as first-class tests.
Next experiment
Test whether adding the last three resolved incidents as context improves recall without increasing hallucination rate.