{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# DHDL Loan Default \u2014 v3 solution (v1 and v2 kept as backups)\nSaurav Atri \u00b7 seeds 42, 7, 2026 \u00b7 5-fold stratified CV. Models were trained offline; the results below are the recorded outputs of that run."
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Full v3 pipeline code\nPlace `Train_set..csv`, `Test_set..csv` and `Sample_Submission..csv` in the data folder, run the training script once per model and seed, then run the stacking script."
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "\"\"\"DHDL Loan Default v3 \u2014 full solution: features + models (v3_train.py) and stacking (v3_stack.py).\nRun: python v3_train.py <lgb|lgbw|xgb|cb|et|mlp> <seed>  for each model/seed, then python v3_stack.py\"\"\"\n# ===== v3_train.py =====\n\"\"\"DHDL Loan Default v3 \u2014 v2 features + repayment-schedule rebuild + in-fold target encoding + diverse models.\nUsage: python v3_train.py <model> [seed]   model in {lgb, xgb, cb, et, mlp, lgbw}\nv1/v2 files are never touched; outputs go to /tmp/ld/v3_*.npy\"\"\"\nimport sys, time, numpy as np, pandas as pd\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import accuracy_score, roc_auc_score\n\nU = \"/tmp/ld/\"\nwhich = sys.argv[1]; SEED = int(sys.argv[2]) if len(sys.argv) > 2 else 42\ntr = pd.read_csv(U + \"Train_set..csv\"); te = pd.read_csv(U + \"Test_set..csv\"); y = tr[\"default\"].values\nCATS = [\"loan_grade\", \"home_ownership\", \"income_verification_status\", \"loan_purpose\", \"state_code\", \"application_type\", \"loan_subgrade\"]\n\ndef fe(d):\n    d = d.copy()\n    d[\"term_m\"] = d[\"loan_term\"].str.extract(r\"(\\d+)\").astype(float)[0] * 12\n    d[\"job_exp_num\"] = d[\"job_experience\"].map({\"<5 Years\": 2.5, \"6-10 years\": 8, \"10+ years\": 12})\n    d[\"job_exp_missing\"] = d[\"job_experience\"].isna().astype(int)\n    d[\"bal_missing\"] = d[\"total_current_balance\"].isna().astype(int)\n    d[\"lwp_missing\"] = d[\"last_week_pay\"].isna().astype(int)\n    d[\"subgrade_num\"] = d[\"loan_subgrade\"].map(lambda s: (ord(s[0]) - 65) * 5 + int(s[1]) if isinstance(s, str) else np.nan)\n    d[\"int_per_loan\"] = d[\"interest_receive\"] / d[\"loan_amnt\"]\n    d[\"pay_progress\"] = d[\"last_week_pay\"] / d[\"term_m\"]\n    d[\"exp_interest\"] = d[\"loan_amnt\"] * d[\"interest_rate\"] / 100 / 12 * d[\"last_week_pay\"].fillna(0) / 4.33\n    d[\"int_gap\"] = d[\"interest_receive\"] - d[\"exp_interest\"]\n    d[\"int_per_week\"] = d[\"interest_receive\"] / (d[\"last_week_pay\"] + 1)\n    d[\"revol_util\"] = d[\"revolving_balance\"] / (d[\"total_revolving_limit\"] + 1)\n    d[\"loan_to_income\"] = d[\"loan_amnt\"] / (d[\"annual_income\"] + 1)\n    r = d[\"interest_rate\"] / 1200\n    d[\"installment\"] = d[\"loan_amnt\"] * r / (1 - (1 + r) ** (-d[\"term_m\"]))\n    d[\"inst_to_income\"] = d[\"installment\"] * 12 / (d[\"annual_income\"] + 1)\n    d[\"int_ratio_expected\"] = d[\"interest_receive\"] / (d[\"exp_interest\"] + 1)\n    d[\"total_expected_int\"] = d[\"installment\"] * d[\"term_m\"] - d[\"loan_amnt\"]\n    d[\"int_share_received\"] = d[\"interest_receive\"] / (d[\"total_expected_int\"].abs() + 1)\n    d[\"weeks_left\"] = d[\"term_m\"] * 4.33 - d[\"last_week_pay\"]\n    d[\"bal_to_income\"] = d[\"total_current_balance\"] / (d[\"annual_income\"] + 1)\n    d[\"revol_to_loan\"] = d[\"revolving_balance\"] / (d[\"loan_amnt\"] + 1)\n    d[\"rate_x_progress\"] = d[\"interest_rate\"] * d[\"pay_progress\"]\n    # ---- v3: exact amortisation schedule rebuild (target-free) ----\n    k = (d[\"last_week_pay\"].fillna(0) / 4.33).clip(lower=0)            # months elapsed\n    k = np.minimum(k, d[\"term_m\"])\n    g = (1 + r) ** k\n    d[\"sched_balance\"] = d[\"loan_amnt\"] * g - d[\"installment\"] * (g - 1) / r   # balance if every payment made\n    d[\"sched_int_paid\"] = d[\"installment\"] * k - (d[\"loan_amnt\"] - d[\"sched_balance\"])\n    d[\"int_vs_sched\"] = d[\"interest_receive\"] / (d[\"sched_int_paid\"] + 1)\n    d[\"int_minus_sched\"] = d[\"interest_receive\"] - d[\"sched_int_paid\"]\n    d[\"implied_months_paid\"] = d[\"interest_receive\"] / (d[\"loan_amnt\"] * r + 1e-6)\n    d[\"months_gap\"] = k - d[\"implied_months_paid\"]\n    d[\"sched_bal_share\"] = d[\"sched_balance\"] / d[\"loan_amnt\"]\n    d[\"lwp_mod\"] = d[\"last_week_pay\"] % 4.33\n    d[\"amnt_round\"] = (d[\"loan_amnt\"] % 1000 == 0).astype(int)\n    return d\n\nfull = pd.concat([fe(tr), fe(te)], ignore_index=True)\nfor gcol in [\"loan_subgrade\", \"state_code\", \"loan_purpose\"]:\n    for c in [\"int_per_loan\", \"pay_progress\", \"interest_rate\", \"loan_to_income\", \"months_gap\"]:\n        full[f\"{c}_vs_{gcol}\"] = full[c] - full.groupby(gcol)[c].transform(\"mean\")\n    full[f\"freq_{gcol}\"] = full.groupby(gcol)[gcol].transform(\"count\")\nfull[\"sg_term\"] = full[\"loan_subgrade\"].astype(str) + \"_\" + full[\"term_m\"].astype(str)\nfull[\"freq_amnt\"] = full.groupby(\"loan_amnt\")[\"loan_amnt\"].transform(\"count\")\nTE_COLS = [\"loan_subgrade\", \"state_code\", \"loan_purpose\", \"sg_term\"]\nfor c in CATS: full[c] = full[c].fillna(\"NA\").astype(\"category\")\nfull = full.drop(columns=[\"ID\", \"loan_term\", \"job_experience\", \"default\"], errors=\"ignore\")\nX0 = full.iloc[:len(tr)].reset_index(drop=True); XT0 = full.iloc[len(tr):].reset_index(drop=True)\n\ndef add_te(Xa, ya, others, seed):\n    \"\"\"Smoothed target means: out-of-fold for training rows, full-train mapping for other frames. No leakage.\"\"\"\n    Xa = Xa.copy(); others = [o.copy() for o in others]; prior = ya.mean(); m = 50\n    for c in TE_COLS:\n        ka = Xa[c].astype(str).values; col = np.zeros(len(Xa))\n        for a, b in StratifiedKFold(5, shuffle=True, random_state=seed).split(Xa, ya):\n            s = pd.DataFrame({\"k\": ka[a], \"y\": ya[a]}).groupby(\"k\")[\"y\"].agg([\"sum\", \"count\"])\n            mp = (s[\"sum\"] + prior * m) / (s[\"count\"] + m)\n            col[b] = pd.Series(ka[b]).map(mp).fillna(prior).values\n        Xa[f\"te_{c}\"] = col\n        s = pd.DataFrame({\"k\": ka, \"y\": ya}).groupby(\"k\")[\"y\"].agg([\"sum\", \"count\"]); mp = (s[\"sum\"] + prior * m) / (s[\"count\"] + m)\n        for o in others: o[f\"te_{c}\"] = o[c].astype(str).map(mp).astype(float).fillna(prior).values\n    return (Xa, *others)\n\ndef prep(d, cat=True):\n    d = d.drop(columns=[\"sg_term\"])\n    if cat: return d\n    d = d.copy()\n    for c in CATS: d[c] = d[c].cat.codes\n    return d\n\ndef fit_predict(Xa, ya, Xb, yb, XT):\n    if which in (\"lgb\", \"lgbw\"):\n        import lightgbm as lgb\n        P = dict(n_estimators=6000, learning_rate=0.02, num_leaves=127, min_child_samples=30, subsample=0.8, subsample_freq=1,\n                 colsample_bytree=0.5, reg_lambda=3.0, n_jobs=4, random_state=SEED, verbose=-1)\n        if which == \"lgbw\": P.update(num_leaves=255, max_depth=12, colsample_bytree=0.35, min_child_samples=15, learning_rate=0.015)\n        m = lgb.LGBMClassifier(**P).fit(prep(Xa), ya, eval_set=[(prep(Xb), yb)], callbacks=[lgb.early_stopping(300, verbose=False)])\n        return m.predict_proba(prep(Xb))[:, 1], m.predict_proba(prep(XT))[:, 1]\n    if which == \"xgb\":\n        import xgboost as xgb\n        m = xgb.XGBClassifier(n_estimators=6000, learning_rate=0.03, max_depth=8, n_jobs=4, subsample=0.8, colsample_bytree=0.5,\n                              min_child_weight=3, reg_lambda=3, enable_categorical=True, tree_method=\"hist\", early_stopping_rounds=300,\n                              random_state=SEED, eval_metric=\"logloss\", max_cat_to_onehot=1)\n        m.fit(prep(Xa), ya, eval_set=[(prep(Xb), yb)], verbose=False)\n        return m.predict_proba(prep(Xb))[:, 1], m.predict_proba(prep(XT))[:, 1]\n    if which == \"cb\":\n        from catboost import CatBoostClassifier\n        def c(d):\n            d = prep(d).copy()\n            for k in CATS: d[k] = d[k].astype(str)\n            return d\n        m = CatBoostClassifier(iterations=3000, learning_rate=0.06, depth=7, l2_leaf_reg=5, random_seed=SEED, verbose=0,\n                               od_type=\"Iter\", od_wait=150, cat_features=CATS, thread_count=4)\n        m.fit(c(Xa), ya, eval_set=(c(Xb), yb))\n        return m.predict_proba(c(Xb))[:, 1], m.predict_proba(c(XT))[:, 1]\n    if which == \"et\":\n        from sklearn.ensemble import ExtraTreesClassifier\n        f = lambda d: prep(d, False).fillna(-999)\n        m = ExtraTreesClassifier(500, min_samples_leaf=3, max_features=0.4, n_jobs=4, random_state=SEED).fit(f(Xa), ya)\n        return m.predict_proba(f(Xb))[:, 1], m.predict_proba(f(XT))[:, 1]\n    if which == \"mlp\":\n        from sklearn.neural_network import MLPClassifier\n        from sklearn.preprocessing import QuantileTransformer\n        f = lambda d: prep(d, False).fillna(prep(Xa, False).median(numeric_only=True))\n        q = QuantileTransformer(output_distribution=\"normal\", random_state=SEED).fit(f(Xa))\n        m = MLPClassifier((256, 128, 64), alpha=1e-3, batch_size=512, learning_rate_init=1e-3, max_iter=60, early_stopping=True,\n                          n_iter_no_change=6, random_state=SEED).fit(q.transform(f(Xa)), ya)\n        return m.predict_proba(q.transform(f(Xb)))[:, 1], m.predict_proba(q.transform(f(XT)))[:, 1]\n\noof = np.zeros(len(X0)); tp = np.zeros(len(XT0)); t = time.time()\nfor a, b in StratifiedKFold(5, shuffle=True, random_state=SEED).split(X0, y):\n    Xa, Xb, XT = add_te(X0.iloc[a].reset_index(drop=True), y[a], [X0.iloc[b].reset_index(drop=True), XT0], SEED)\n    pv, pt = fit_predict(Xa, y[a], Xb, y[b], XT); oof[b] = pv; tp += pt / 5\n    print(which, SEED, \"fold\", round(time.time() - t), flush=True)\nprint(f\"{which} s{SEED}: acc@.5={accuracy_score(y, oof > .5):.5f} auc={roc_auc_score(y, oof):.5f} {time.time()-t:.0f}s\", flush=True)\nnp.save(f\"{U}v3_{which}_s{SEED}_oof.npy\", oof); np.save(f\"{U}v3_{which}_s{SEED}_test.npy\", tp)\n\n# ===== v3_stack.py =====\n\"\"\"v3 stacking (logistic + LightGBM meta, hill-climb weights), nested cut-off, decision rule vs v2. Never overwrites v1/v2.\"\"\"\nimport json, glob, numpy as np, pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import accuracy_score, roc_auc_score, confusion_matrix\n\nD = \"/tmp/ld/\"; R = \"/dev-server/\"\nV2 = json.load(open(R + \"src/data/loan-default-v2-evidence.json\")); V2ACC, V2NEST = V2[\"final\"][\"cvAccuracy\"], V2[\"final\"][\"nestedAccuracy\"]\ntr = pd.read_csv(D + \"Train_set..csv\"); te = pd.read_csv(D + \"Test_set..csv\"); ss = pd.read_csv(D + \"Sample_Submission..csv\"); y = tr[\"default\"].values\nNAMES = {\"lgb\": \"LightGBM v3 \u00b7 127 leaves\", \"lgbw\": \"LightGBM v3 \u00b7 255 leaves, deep\", \"xgb\": \"XGBoost v3\", \"cb\": \"CatBoost v3\",\n         \"et\": \"ExtraTrees v3\", \"mlp\": \"Neural network (tabular MLP)\"}\noof, tst, per_seed = {}, {}, {}\nfor k in NAMES:\n    f = sorted(glob.glob(f\"{D}v3_{k}_s*_oof.npy\"))\n    if not f: continue\n    oof[k] = np.mean([np.load(x) for x in f], 0); tst[k] = np.mean([np.load(x.replace(\"_oof\", \"_test\")) for x in f], 0)\n    per_seed[k] = {int(x.split(\"_s\")[-1].split(\"_\")[0]): round(accuracy_score(y, np.load(x) > .5) * 100, 2) for x in f}\nkeys = list(oof)\nTS = np.arange(0.30, 0.71, 0.005)\ndef thr(p, yy): a = [accuracy_score(yy, p > t) for t in TS]; i = int(np.argmax(a)); return float(TS[i]), a[i]\ndef logit(p): p = np.clip(p, 1e-6, 1 - 1e-6); return np.log(p / (1 - p))\nZ = np.column_stack([logit(oof[k]) for k in keys]); ZT = np.column_stack([logit(tst[k]) for k in keys])\ncands = {k: (oof[k], tst[k]) for k in keys}\n\n# hill-climbing weights (Caruana) \u2014 chosen out-of-fold on a second split to keep it honest\ndef hill(P, yy, iters=40):\n    w = np.zeros(P.shape[1]); best = -1\n    for _ in range(iters):\n        sc = []\n        for j in range(P.shape[1]):\n            w2 = w.copy(); w2[j] += 1; p = P @ w2 / w2.sum(); sc.append(roc_auc_score(yy, p))\n        j = int(np.argmax(sc))\n        if sc[j] <= best: break\n        best = sc[j]; w[j] += 1\n    return w / w.sum()\nP = np.column_stack([oof[k] for k in keys]); PT = np.column_stack([tst[k] for k in keys])\nhc = np.zeros(len(y))\nfor a, b in StratifiedKFold(5, shuffle=True, random_state=2026).split(P, y): hc[b] = P[b] @ hill(P[a], y[a])\nW = hill(P, y); cands[\"hill\"] = (hc, PT @ W)\n\nlr = np.zeros(len(y))\nfor a, b in StratifiedKFold(5, shuffle=True, random_state=2026).split(Z, y):\n    lr[b] = LogisticRegression(C=1.0, max_iter=2000).fit(Z[a], y[a]).predict_proba(Z[b])[:, 1]\ncands[\"stack\"] = (lr, LogisticRegression(C=1.0, max_iter=2000).fit(Z, y).predict_proba(ZT)[:, 1])\n\nimport lightgbm as lgb\nraw = tr[[\"interest_rate\", \"loan_amnt\", \"last_week_pay\"]].values; rawT = te[[\"interest_rate\", \"loan_amnt\", \"last_week_pay\"]].values\nM = np.column_stack([Z, raw]); MT = np.column_stack([ZT, rawT]); gm = np.zeros(len(y)); gt = np.zeros(len(te))\nmp = dict(n_estimators=400, learning_rate=0.03, num_leaves=15, min_child_samples=200, subsample=0.8, subsample_freq=1, n_jobs=4, verbose=-1, random_state=1)\nfor a, b in StratifiedKFold(5, shuffle=True, random_state=2026).split(M, y):\n    m = lgb.LGBMClassifier(**mp).fit(M[a], y[a]); gm[b] = m.predict_proba(M[b])[:, 1]; gt += m.predict_proba(MT)[:, 1] / 5\ncands[\"stack_lgb\"] = (gm, gt)\ncands[\"stack_blend\"] = ((lr + gm) / 2, (cands[\"stack\"][1] + gt) / 2)\n\nLABEL = {\"hill\": \"Hill-climb weighted blend\", \"stack\": \"Stacked (logistic meta-model)\", \"stack_lgb\": \"Stacked (LightGBM meta-model)\",\n         \"stack_blend\": \"Two-level stack (logistic + LightGBM meta)\"}\nrows = []\nfor k, (p, _) in cands.items():\n    t, a = thr(p, y); nest = []\n    for s in (7, 42, 2026):\n        for a_, b_ in StratifiedKFold(5, shuffle=True, random_state=s).split(p.reshape(-1, 1), y):\n            tt, _ = thr(p[a_], y[a_]); nest.append(accuracy_score(y[b_], p[b_] > tt))\n    rows.append(dict(id=k, name=NAMES.get(k, LABEL.get(k)), acc05=round(accuracy_score(y, p > .5) * 100, 2), auc=round(roc_auc_score(y, p), 4),\n                     threshold=round(t, 3), accTuned=round(a * 100, 2), accNested=round(float(np.mean(nest)) * 100, 2)))\nrows.sort(key=lambda r: -r[\"accNested\"]); print(pd.DataFrame(rows).to_string())\nbest = rows[0]; p, pt = cands[best[\"id\"]]; t = best[\"threshold\"]\npred = (p > t).astype(int); tn, fp, fn, tpv = confusion_matrix(y, pred).ravel()\ngain_nested = round(best[\"accNested\"] - V2NEST, 2); promoted = gain_nested >= 0.10\n\n# error analysis of the final model\nerr = pd.DataFrame({\"grade\": tr[\"loan_grade\"], \"term\": tr[\"loan_term\"], \"wrong\": pred != y, \"near\": np.abs(p - t) < 0.1})\nerrors = dict(totalWrong=int((pred != y).sum()), nearCutoffShare=round(float(err.loc[err.wrong, \"near\"].mean()) * 100, 1),\n              byGrade=[dict(group=g, errRate=round(float(v) * 100, 2)) for g, v in err.groupby(\"grade\")[\"wrong\"].mean().items()],\n              byTerm=[dict(group=g, errRate=round(float(v) * 100, 2)) for g, v in err.groupby(\"term\")[\"wrong\"].mean().items()])\n\nsub = pd.DataFrame({\"ID\": te[\"ID\"], \"default\": (pt > t).astype(int)})\nassert len(sub) == 39933 and set(sub[\"default\"].unique()) <= {0, 1} and (sub[\"ID\"].values == ss[\"ID\"].values).all()\nsub.to_csv(R + \"public/submission/DHDL_Loan_Default_Submission_v3.csv\", index=False)\nv2s = pd.read_csv(R + \"public/submission/DHDL_Loan_Default_Submission_v2.csv\")\nadv = json.load(open(D + \"adversarial.json\")) if glob.glob(D + \"adversarial.json\") else None\nev = dict(version=\"v3\", v2Accuracy=V2ACC, v2Nested=V2NEST, v1Accuracy=V2[\"v1Accuracy\"], models=rows, perSeedAcc05=per_seed,\n          hillWeights={k: round(float(w), 3) for k, w in zip(keys, W)},\n          final=dict(id=best[\"id\"], name=best[\"name\"], threshold=t, cvAccuracy=best[\"accTuned\"], nestedAccuracy=best[\"accNested\"], auc=best[\"auc\"],\n                     gainVsV2=round(best[\"accTuned\"] - V2ACC, 2), gainNestedVsV2=gain_nested, promoted=promoted,\n                     confusion=dict(tn=int(tn), fp=int(fp), fn=int(fn), tp=int(tpv)), testDefaultRate=round(float(sub[\"default\"].mean()) * 100, 2),\n                     agreementWithV2=round(float((v2s[\"default\"].values == sub[\"default\"].values).mean()) * 100, 2), rows=len(sub)),\n          errors=errors, adversarial=adv,\n          seeds=sorted({int(x.split(\"_s\")[-1].split(\"_\")[0]) for x in glob.glob(D + \"v3_*_oof.npy\")}),\n          newFeatures=[\"rebuilt repayment schedule: expected balance and interest if every payment were made\",\n                       \"interest received vs schedule (ratio and gap)\", \"implied months paid vs months elapsed\",\n                       \"in-fold smoothed default rates for sub-grade, state, purpose, sub-grade \u00d7 term\",\n                       \"peer comparisons of months-behind within sub-grade, state, purpose\", \"loan-amount frequency and round-amount flag\"])\njson.dump(ev, open(R + \"src/data/loan-default-v3-evidence.json\", \"w\"), indent=2)\nprint(json.dumps(ev[\"final\"], indent=1))\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Recorded results (v3)\n| Model | Acc @0.5 | Tuned | Honest | AUC |\n|---|---|---|---|---|\n| Stacked (logistic meta-model) | 91.85% | 91.87% | 91.83% | 0.9645 |\n| Stacked (LightGBM meta-model) | 91.84% | 91.85% | 91.83% | 0.9646 |\n| Two-level stack (logistic + LightGBM meta) | 91.86% | 91.87% | 91.83% | 0.9648 |\n| LightGBM v3 \u00b7 255 leaves, deep | 91.77% | 91.83% | 91.81% | 0.9637 |\n| Hill-climb weighted blend | 91.77% | 91.83% | 91.81% | 0.9638 |\n| LightGBM v3 \u00b7 127 leaves | 91.71% | 91.75% | 91.71% | 0.9632 |\n| CatBoost v3 | 91.6% | 91.71% | 91.69% | 0.9617 |\n| XGBoost v3 | 91.51% | 91.6% | 91.6% | 0.9606 |\n| ExtraTrees v3 | 91.13% | 91.33% | 91.31% | 0.9583 |\n| Neural network (tabular MLP) | 90.16% | 90.17% | 90.12% | 0.9397 |\n\nFinal: **Stacked (logistic meta-model)**, cut-off 0.49, CV accuracy **91.87%**, honest estimate 91.83%, +0.72 pts over v2 (91.15%) and v1 (89.60%). Train/test difference check: 0.4996 (0.5 = identical)."
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}