Model Comparison & Final Model
What you'll see here
Which algorithm scores highest on accuracy, how it was tuned, and what drives its predictions.
CV accuracy
89.6%
threshold 0.38
ROC-AUC
0.9343
Precision · Recall
80.56% · 74.1%
Final model
LightGBM
LightGBM · 255 leaves
Out-of-fold results (threshold 0.5)
| Model | CV accuracy | AUC | Train accuracy | Gap |
|---|---|---|---|---|
| LightGBM · 127 leaves | 89.03% | 0.9335 | 96.24% | 7.21 pts |
| LightGBM · 255 leaves | 89.03% | 0.9343 | 97.29% | 8.26 pts |
| LightGBM (base) | 88.89% | 0.9307 | 95.79% | 6.9 pts |
| LightGBM · 31 leaves | 88.75% | 0.9295 | 94.81% | 6.07 pts |
| LightGBM · 63 leaves, dense | 88.7% | 0.9288 | 95.94% | 7.24 pts |
| XGBoost | 88.42% | 0.9254 | 95.58% | 7.17 pts |
| Random Forest | 84.98% | 0.9022 | 99.25% | 14.27 pts |
| Logistic Regression | 81.86% | 0.8301 | 81.91% | 0.05 pts |
Tuning then moved the decision threshold from 0.5 to 0.38, chosen on out-of-fold predictions to maximise accuracy: 89.6%. A weighted blend with XGBoost was tested and gave no gain, so the single LightGBM was kept. Because the threshold was picked on the same folds, expect the leaderboard score to land slightly below the CV figure.
Confusion matrix (out-of-fold, 93,174 loans)
| Pred. repaid | Pred. default | |
|---|---|---|
| Actual repaid | 67,087 | 3,958 |
| Actual default | 5,731 | 16,398 |
Top drivers (LightGBM gain)