Home

Data & RAG Pipeline

What you'll see here

What the source document looks like and every setting used to prepare it for retrieval.

Pages
4,114
Tokens
3.08M
cl100k_base
Chunks
11,741
512 tok · 64 overlap
Embedding dims
384
Pipeline settings
Dataset
medical_diagnosis_manual.pdf — Merck Manual of Diagnosis & Therapy, 19th ed. (2011)
Loader
PyMuPDFLoader, one document per page
Cleaning
Removed the licence watermark and running header repeated on every page
Splitter
RecursiveCharacterTextSplitter (cl100k_base), chunk_size 512, chunk_overlap 64
Embedding
BAAI/bge-small-en-v1.5 (384 dimensions, normalised)
Vector DB
Chroma, cosine distance, persisted to disk
Retriever
Similarity search, k = 3, plus k results for a textbook-style rewrite of the question
Generator
gpt-4o-mini, temperature 0, top_p 0.95, max_tokens 512
Why a question rewrite?

Scenario wording pulls a pure similarity search off course: "fractured their leg during a hiking trip" matched pages about amputation and walking aids instead of the fractures chapter.

Each question is therefore also rewritten into the section title a textbook would use (for example "Leg Fractures: diagnosis and treatment"). Results for both are merged and de-duplicated.