ACCEPTEDDraft2025-04-20 — updated 2025-07-10
ADR-002: Retrieval over fine-tuning for runbook answers
Answer operational questions by retrieving from runbooks rather than fine-tuning a model on internal incidents.
Launch stateThese entries are placeholders while the real case studies and experiment write-ups are prepared for publication.
Context
- On-call engineers needed faster answers to known operational questions without relying on undocumented tribal knowledge.
- Fine-tuning on incident history risked baking outdated or sensitive details into model weights.
Decision
- Use retrieval over indexed runbooks and incident notes to ground answers.
- Require every answer to cite a source or refuse explicitly.
Alternatives considered
- Fine-tune a small model on incident history. Rejected because updates are expensive and citations are unreliable.
- Use a general model without grounding. Rejected because it invents steps for procedural questions.
Consequences
- Answers stay tied to current documents; updating a runbook immediately changes what the system can say.
- Evaluation must include refusal cases as first-class tests.
- Chunking and embedding quality become the main engineering work, not model training.