Evaluation & Tuning
What you'll see here
A separate judge model (gpt-4o) scores every answer 1–5 against the same manual excerpts. A rule-based clinical checklist gives a second view.
Mean judge scores by approach
RAG scores per question
| Question | Groundedness | Relevance |
|---|---|---|
| Sepsis in critical care | 5 | 5 |
| Appendicitis | 5 | 5 |
| Patchy hair loss | 5 | 5 |
| Brain tissue injury | 5 | 5 |
| Leg fracture while hiking | 5 | 4 |
| Mean | 5.0 | 4.8 |
Clinical checklist coverage
| Question | Base LLM | Prompt-Engineered | RAG |
|---|---|---|---|
| Sepsis in critical care | 6/6 | 6/6 | 4/6 |
| Appendicitis | 5/7 | 7/7 | 6/7 |
| Patchy hair loss | 5/5 | 5/5 | 5/5 |
| Brain tissue injury | 3/6 | 5/6 | 5/6 |
| Leg fracture while hiking | 6/7 | 7/7 | 6/7 |
| Coverage | 80.6% | 96.8% | 83.9% |
Counts key clinical points the manual says a complete answer should include. It rewards breadth, not whether a statement is supported by the source.
Six RAG configurations
| Configuration | Groundedness | Relevance | Combined | Context tokens |
|---|---|---|---|---|
| C1 baseline (512 tok, k=3, T=0) | 5 | 4.8 | 9.8 | 1,790 |
| C2 more context (k=6) | 5 | 4.8 | 9.8 | 3,409 |
| C3 less context (k=1) | 4.8 | 4.6 | 9.399999999999999 | 827 |
| C4 creative (T=0.7, top_p=0.9) | 4.8 | 4.8 | 9.6 | 1,790 |
| C5 short answer (max_tokens=200) | 4.8 | 4.4 | 9.2 | 1,790 |
| C6 large chunks (1000 tok, k=3) | 5 | 4.8 | 9.8 | 3,348 |