Home

Model Comparison & Final Model

What you'll see here

Which algorithm scores highest on accuracy, how it was tuned, and what drives its predictions.

CV accuracy
89.6%
threshold 0.38
ROC-AUC
0.9343
Precision · Recall
80.56% · 74.1%
Final model
LightGBM
LightGBM · 255 leaves
Out-of-fold results (threshold 0.5)
ModelCV accuracyAUCTrain accuracyGap
LightGBM · 127 leaves89.03%0.933596.24%7.21 pts
LightGBM · 255 leaves89.03%0.934397.29%8.26 pts
LightGBM (base)88.89%0.930795.79%6.9 pts
LightGBM · 31 leaves88.75%0.929594.81%6.07 pts
LightGBM · 63 leaves, dense88.7%0.928895.94%7.24 pts
XGBoost88.42%0.925495.58%7.17 pts
Random Forest84.98%0.902299.25%14.27 pts
Logistic Regression81.86%0.830181.91%0.05 pts

Tuning then moved the decision threshold from 0.5 to 0.38, chosen on out-of-fold predictions to maximise accuracy: 89.6%. A weighted blend with XGBoost was tested and gave no gain, so the single LightGBM was kept. Because the threshold was picked on the same folds, expect the leaderboard score to land slightly below the CV figure.

Confusion matrix (out-of-fold, 93,174 loans)
Pred. repaidPred. default
Actual repaid67,0873,958
Actual default5,73116,398
Top drivers (LightGBM gain)