Home

Evaluation & Tuning

What you'll see here

A separate judge model (gpt-4o) scores every answer 1–5 against the same manual excerpts. A rule-based clinical checklist gives a second view.

Mean judge scores by approach
RAG scores per question
QuestionGroundednessRelevance
Sepsis in critical care55
Appendicitis55
Patchy hair loss55
Brain tissue injury55
Leg fracture while hiking54
Mean5.04.8
Clinical checklist coverage
QuestionBase LLMPrompt-EngineeredRAG
Sepsis in critical care6/66/64/6
Appendicitis5/77/76/7
Patchy hair loss5/55/55/5
Brain tissue injury3/65/65/6
Leg fracture while hiking6/77/76/7
Coverage80.6%96.8%83.9%

Counts key clinical points the manual says a complete answer should include. It rewards breadth, not whether a statement is supported by the source.

Six RAG configurations
ConfigurationGroundednessRelevanceCombinedContext tokens
C1 baseline (512 tok, k=3, T=0)54.89.81,790
C2 more context (k=6)54.89.83,409
C3 less context (k=1)4.84.69.399999999999999827
C4 creative (T=0.7, top_p=0.9)4.84.89.61,790
C5 short answer (max_tokens=200)4.84.49.21,790
C6 large chunks (1000 tok, k=3)54.89.83,348