Submission & Insights
The files to upload to the hackathon, and what DHDL should do with the findings.
Leaderboard: v9 scored 94.37558% against the leader's 94.40563%, a gap of about 12 loans. v10 adds a full-data XGBoost model and a small second-opinion model for loans close to the cut-off. The second opinion was the only v8 idea that gained on all three splits (+0.045 / +0.003 / +0.014 points). It changes 206 test loans and agrees with v9 on 99.64% of loans.
Same signals, blend and cut-offs as v6, but each final model learns from all training loans instead of 80%, and 8 LightGBM plus 4 CatBoost runs are averaged. Cross-validation can't measure this change, because it only affects how the final models are trained, so the leaderboard is the test. It agrees with v6 on 99.6% of loans and predicts 24.63% defaults.
The model is still wrong on 5.43% of training loans. Only 34.9% of those mistakes are close calls; the rest are loans the data makes look clearly safe or clearly risky but which ended the other way. Mistakes rise from 3.33% for grade A to 11.07% for grade G. No idea beat the v6 recipe on all three splits, so no v8 file was made.
| Idea (honest estimate) | Split 42 | Split 7 | Split 2026 |
|---|---|---|---|
| Five LightGBM seeds instead of two | 94.527% (+0.01) | 94.529% (-0.04) | 94.536% (-0.02) |
| Add ExtraTrees to the mix | 94.524% (+0.00) | 94.55% (-0.02) | 94.57% (+0.02) |
| Cut-off matched to the test default share | 94.51% (-0.01) | 94.547% (-0.02) | 94.545% (-0.01) |
| Near-cut-off specialist model | 94.566% (+0.04) | 94.574% (+0.00) | 94.566% (+0.01) |
| How common each exact "last week paid" value is (charge-off batch marker) | 94.508% (-0.00) | 94.534% (-0.00) | 94.505% (-0.01) |
| Interest a healthy loan would have paid, from a model fitted on non-default loans only | 94.5% (-0.01) | 94.515% (-0.02) | 94.496% (-0.01) |
Round 3 AI advisors (v8) — full advice
1. **CatBoost with native categorical interactions.** Train ordered boosting, depth 5–8, on the existing features plus native subgrade, purpose, state, verification and ownership; blend with LightGBM. Different categorical handling could correct complementary errors. **Expected gain:** +0.04–0.18 accuracy points. **Leakage:** low. 2. **Structurally different XGBoost.** Try `hist`, depth 3–5, conservative `min_child_weight`, and strong L2; blend rather than replace LightGBM. Shallow depth-wise trees may generalise differently from leaf-wise growth around ambiguous repayment patterns. **Expected gain:** +0.02–0.12 points. **Leakage:** low. 3. **Feature-family dropout ensemble.** Train variants excluding, separately, raw ID, pricing/cohort features, and borrower-credit features while retaining repayment reconstructions; select blend weights on inner OOF predictions. Removing shortcut families can create more useful diversity than additional seeds. **Expected gain:** +0.02–0.10 points. **Leakage:** low with nested weight selection. 4. **Hierarchically smoothed categorical target encoding.** Encode `(purpose, verification, ownership)` and `(subgrade, application_type, job_experience)`, shrinking sparse groups toward pairwise and marginal rates; cross-fit inside each training fold. These capture borrower-segment priors rather than neighbourhood effects. **Expected gain:** +0.01–0.10 points. **Leakage:** medium—ordinary full-training encodings invalidate validation. 5. **Measurement/provenance fingerprints.** Extract raw-string decimal precision, amount divisibility, zero-versus-missing flags, and joint missingness patterns; cross these with broad ID bands. Different servicing/export regimes may change what the same repayment feature means. **Expected gain:** +0.01–0.08 points. **Leakage:** medium—source shortcuts can fail across time. 6. **Credit-capacity ratios, not payment-history proxies.** Add revolving utilisation, unused revolving capacity divided by income, revolving share of current balance, and delinquency/public-record counts per account; include denominator-zero flags. They may separate genuine distress from benign repayment irregularity. **Expected gain:** +0.00–0.07 points. **Leakage:** low. 7. **Conservative label-noise downweighting.** Within each outer training fold, identify loans assigned <2% probability to their recorded label by multiple inner-OOF model families; test weights of 0.5 rather than deletion, affecting at most 1% of training rows. This may stop contradictory labels distorting otherwise clean boundaries. **Expected gain:** −0.03–+0.08 points. **Leakage:** medium; also risks suppressing legitimate rare cases. 8. **Covariate-shift importance weighting.** Train a shallow train-versus-test classifier on label-free features; convert cross-fitted probabilities to density-ratio weights, clip to `[0.5, 2]`, and compare weighted/unweighted blends. Useful only if test composition differs in regions with training support. **Expected gain:** −0.03–+0.07 points. **Leakage:** medium—label-free test use must be permitted, and benefits need shift-matched validation.
# 8 untried ideas, ranked by expected gain
Gains are estimates against your honest 94.55% baseline. Only idea 1 plausibly reaches +0.3. The rest are realistically +0.03 to +0.15 each and only partly additive.
1. **Match rows to the public LendingClub loan file (only if the rules allow external data).**
- *Why:* The columns look like renamed LendingClub fields. A fuzzy join on loan_amnt, interest_rate, annual_income, state_code, dti, total_acc and revolving_balance could recover the true loan status and issue date for many rows.
- *Gain:* +0.5 to +5.
- *Leakage:* **High.** It may breach the rules, so check them first. Even if you don't submit it, you can use it offline to see which rows you misclassify and why.
2. **Use the fractional part of implied k as partial-payment evidence.**
- *Why:* Your v5 k is continuous. Recompute the exact cent-rounded installment, then measure:
- the distance from k to the nearest whole number;
- the residual against the integer-k cumulative interest.
- Non-integer residuals signal partial payments, late fees or hardship plans, which are typical of loans about to default. This is not the same as k versus age.
- *Gain:* +0.05 to +0.2.
- *Leakage:* Low.
3. **Treat spikes in exact last_week_pay values (counted over train+test) as charge-off batch markers.**
- *Why:* Servicers charge off in batches, so defaulted loans' final payment weeks cluster on specific values.
- *Features:* count of each exact value across train+test; value frequency within ID-neighbourhood windows; z-score against a smooth density.
- *Gain:* +0.05 to +0.15.
- *Leakage:* Low. No labels are used; test rows are used transductively.
4. **Estimate loan age on the merged train+test ID sequence (if you aren't already).**
- *Why:* About 30% of the ID slots are test rows. Including them in the rolling p90 window makes it denser. That cuts noise in age and weeks-behind estimates, and you said feature noise is the limiting factor.
- *Gain:* +0.03 to +0.1.
- *Leakage:* Low.
5. **Clean label noise with confident learning, then retrain.**
- *Why:* With 5.5% error and no duplicates, part of the remaining error may be mislabelled or inherently ambiguous rows.
- *Method:* Flag train rows that out-of-fold predictions confidently contradict (p > 0.97 against the label). Drop them or down-weight them to 0.3, then retrain.
- *Note:* Validate only on uncleaned folds.
- *Gain:* +0.03 to +0.1.
- *Leakage:* Low, provided cleaning happens inside the folds.
6. **Stack models trained on different feature sets with a level-2 logistic regression.**
- *Why:* This is diversity from feature views, not seeds:
- a v4-only model;
- a v5-only model;
- a v6-only model;
- a CatBoost model on raw columns plus subgrade×state and grade×purpose categoricals with ordered target statistics.
- Their errors are likely to be less correlated than seed variants.
- *Gain:* +0.03 to +0.1.
- *Leakage:* Low, if the out-of-fold predictions are built cleanly.
7. **Fit separate models per loan_term (36/60), crossed with loan-age tercile.**
- *Why:* The amortisation shape and the meaning of weeks-behind differ by term and stage, so one global tree set compromises across regimes.
- *Method:* Blend the segment models with the global model rather than replacing it.
- *Gain:* +0.02 to +0.08.
- *Leakage:* Low.
8. **Reweight training rows using adversarial validation.**
- *Why:* The ID/time split means the test set may sit at a different vintage mix.
- *Method:* Train a train-vs-test classifier and weight train rows by p/(1−p). Also check whether the useful age features drift between train and test.
- *Gain:* +0.02 to +0.06. More if drift exists, around 0 if not.
- *Leakage:* Low.
**Recommended order given 3 days:**
- Day 1: test ideas 2, 3 and 4. They are cheap, label-free, and target feature noise.
- Day 2: run 5 and 6.
- Day 3: lock in the final ensemble.
- Decide on idea 1 today, based on the rules.1. **Healthy-Loan Regressor Residuals** - **Idea**: Train a regressor to predict `interest_receive` *exclusively* on `default == 0` rows, then use its residuals as a feature for the main classifier. - **Why it could work**: Defaults stop accruing interest. A model trained only on healthy loans will severely over-predict interest for defaulted loans, making the residual a massive, clean signal. - **Expected gain**: +0.08 to +0.15 - **Leakage risk**: Low 2. **Exact EMI / PMT Physics** - **Idea**: Calculate exact monthly Equated Monthly Installment (EMI) using standard financial PMT formula on `loan_amnt`, `interest_rate`, and `loan_term`. - **Why it could work**: Trees cannot natively compute complex algebraic formulas. Providing the exact monthly obligation allows the model to accurately cross-reference current balances and interest received against what *should* have been paid. - **Expected gain**: +0.05 to +0.10 - **Leakage risk**: Low 3. **CatBoost Addition for Ensembling** - **Idea**: Train a CatBoost classifier (handles categoricals natively, uses oblivious/symmetric trees) and blend it with your LightGBM/ExtraTrees. - **Why it could work**: CatBoost partitions data entirely differently than LightGBM. Given the highly structured deterministic math in this dataset, symmetric trees often capture rigid interaction rules better, leading to high ensemble weights. (90k rows is fast on CPU). - **Expected gain**: +0.04 to +0.08 - **Leakage risk**: Low 4. **Shallow MLP on Exact Math Features** - **Idea**: Train a small Multi-Layer Perceptron (scikit-learn `MLPClassifier` or basic PyTorch) *only* on your engineered time/amortization/gap math, then stack its OOF predictions. - **Why it could work**: Amortization schedules form curved, diagonal decision boundaries. Tree models require hundreds of inefficient "zigzag" splits to approximate this, whereas an MLP solves it natively and will capture edge cases LightGBM missed. - **Expected gain**: +0.03 to +0.06 - **Leakage risk**: Low 5. **Integer Discretization of Leakage Features** - **Idea**: Explicitly round or floor your engineered continuous float features (`weeks_behind`, implied months `k`) into exact integer bins (-1, 0, 1, 2, 3+). - **Why it could work**: Missing a payment is a discrete, binary state, but continuous math introduces floating-point noise. Forcing integers prevents trees from wasting depth splitting on noise and anchors them to exact "missed payment" states. - **Expected gain**: +0.02 to +0.05 - **Leakage risk**: Low 6. **Adversarial Validation Weighting** - **Idea**: Train a model to predict Train (0) vs Test (1). Use the resulting probability to weight the training samples in your LightGBM default model. - **Why it could work**: Because `ID` is time-ordered, the test set is from the future. Weighting forces the model to heavily prioritize the most recent training loans, adapting to shifting macro conditions or underwriting rules. - **Expected gain**: +0.02 to +0.04 - **Leakage risk**: Low 7. **Fractional / Decimal Modulo Artifacts** - **Idea**: Extract the decimal remainder (`x % 1`) for `interest_receive`, `total_current_balance`, and `loan_amnt`. - **Why it could work**: Bank IT systems often apply specific rounding rules, fee additions, or decimal truncations exactly when a loan shifts from "active" to "default/charge-off" status. - **Expected gain**: +0.01 to +0.04 - **Leakage risk**: High 8. **Expanding Window Target Encoding over Time** - **Idea**: Calculate an expanding mean of `default` sorted by `ID` for interactions like `subgrade` + `state_code` or `loan_purpose`. - **Why it could work**: Standard target encoding loses the temporal drift of risk. An expanding window correctly maps how default rates for specific loan grades in specific states degraded as the timeline approached the test set. - **Expected gain**: +0.01 to +0.03 - **Leakage risk**: Medium
No two training loans share every detail, so identical-looking loans do not cap accuracy (100% in theory). The real limit is that the information we have does not fully separate loans that end up defaulting from loans that don't. None of the six v7 ideas gave a gain that held across splits, so 95% was not reached and no v7 file was made.
| Idea (quick screen, base 94.471%) | Suggested by | Accuracy | Change |
|---|---|---|---|
| Rebuilt payment history (missed payments vs 3 age estimates) | plan | 94.506% | +0.04 |
| Outcomes of the 50 most similar loans (fold-safe) | plan | 94.448% | -0.02 |
| Specialist second model for near-cut-off and F–G loans | plan + Claude | 94.478% | +0.01 |
| Exact-cent schedule residuals, payoff fingerprint, balance vs schedule | GPT-6 Astra, Claude, Gemini | 94.526% | +0.06 |
| Down-weight confidently wrong training loans | Claude | 94.51% | +0.04 |
| Payment history + exact-cent schedule combined | combined | 94.544% | +0.07 |
| Split seed (best idea) | v6 features | v7 features | Change |
|---|---|---|---|
| 2026 | 94.471% | 94.544% | +0.07 |
| 42 | 94.541% | 94.567% | +0.03 |
| 7 | 94.567% | 94.521% | -0.05 |
Round 3 AI advisors — full advice
1. **Link to the original Lending Club records—only if external data are permitted.** Match on amount, rate, income, balances and account counts; require unique, high-confidence matches. Recovering actual issue dates, installments and funded amounts could remove the uncertainty driving your current ceiling. Use only snapshot-appropriate, permitted fields—not later outcomes. **Expected gain: +0.3–1.0 accuracy points, conditional on coverage; leakage risk: high.** 2. **Detect early-payoff and maturity fingerprints explicitly.** Compute full-term scheduled interest, `interest_receive / full_term_interest`, and residuals to candidate early-payoff totals including a final partial payment. A loan that stopped paying because it closed should differ from one that stopped because it defaulted; “weeks behind” can conflate them. **Expected gain: +0.10–0.40 points; leakage risk: low.** 3. **Replace the single amortization inversion with competing cash-flow hypotheses.** Fit ordinary monthly payments, shifted first-payment timing, partial final payments, and extra-principal-payment scenarios; retain each scenario’s best-fit payment count, residual and fit-margin. These can expose systematic failures of your current implied `k`. **Expected gain: +0.08–0.35 points; leakage risk: low.** 4. **Train a genuinely different model: CPU CatBoost, then OOF-blend.** Use native categoricals, explicit missing/sentinel tokens, and your strongest reconstructed features; try both shallow and moderately deep trees. Ordered categorical interactions and different split geometry may correct LightGBM’s errors more efficiently than another large feature batch. **Expected gain: +0.05–0.30 points; leakage risk: low with strictly OOF blend selection.** 5. **Use soft servicing-regime experts instead of one global classifier.** Build separate predictors for immature/current, delinquent-looking, near-maturity and possible-prepayment loans, then blend through feature-only soft gates. The same payment gap can imply very different outcomes across these regimes; this goes beyond per-grade thresholds. **Expected gain: +0.05–0.25 points; leakage risk: low.** 6. **Propagate age uncertainty rather than adding another point estimate.** Bootstrap neighbour/cohort age estimates; supply age dispersion, estimator disagreement, and probabilities of being ≥1/2/3 installments behind. Your strongest feature is latent and noisy—uncertainty can distinguish genuine delinquency from an unreliable age estimate. **Expected gain: +0.04–0.20 points; leakage risk: low if reconstruction is fold-safe.** 7. **Train an OOF error-correction model focused on disagreements.** Generate cross-fitted predictions from v5, v6 and alternative seeds/models; fit a small regularized stacker using logits, prediction disagreement and a few payment-regime features. This can learn *when* the newer reconstruction is worse, rather than assuming every added feature helps every loan. **Expected gain: +0.03–0.18 points; leakage risk: medium—requires nested evaluation to avoid optimistic stacking.** 8. **Optimize the actual accuracy decision rule with shrinkage.** On nested OOF predictions, test a global cutoff plus regularized term × age-regime offsets, retaining offsets only when stable across folds. Your per-grade thresholds do not cover these regimes, and small systematic probability bias near the decision boundary can cost accuracy. **Expected gain: +0.02–0.12 points; leakage risk: medium from threshold overfitting.**
# Round 3: 8 untried ideas, ranked by expected gain No single idea below is likely to add +0.3 by itself. Getting there realistically means stacking #1, #2 and #3, then ensembling. Gains are honest-CV accuracy points on base 94.47. 1. **Exact-cent schedule residual.** Recompute the installment and per-month interest at cent rounding (LC style). Then find the best integer k and add these features: - residual = interest_receive minus cumulative interest at k, - the fractional part of k, - an "exact match" flag. - *Why:* On-time loans match the schedule to the cent. Late, partial or charged-off payments accrue extra daily interest or break the pattern, so they leave non-integer residuals. Your continuous k likely smooths this signal away. - *Gain:* +0.08–0.20. *Leakage:* low. 2. **Payoff vs default disambiguation.** A fully paid early payoff's last payment holds partial-period interest plus no further interest. Build a "payoff signature" (residual is about one partial month, with k ≈ last_week_pay/4.33 and no lag before it) against a "default signature" (k falls behind before stopping). - *Why:* The likely residual noise is stopped-paying loans whose label is either paid-off or defaulted. This feature targets exactly that confusion. - *Gain:* +0.05–0.15. *Leakage:* low. 3. **Same-borrower partial-key matching.** Match train and test rows on borrower attributes: - Keys: annual_income, total_acc, total_revolving_limit, total_current_balance, state, home_ownership, job_experience. - Use a tolerance for snapshot drift. - Features: the other loan's default label (fold-safe) and its k/lag. - *Why:* There are no exact row duplicates, but repeat borrowers or co-applicants share profiles, and their behaviour correlates strongly. - *Gain:* +0.03–0.15, depending on match rate (check that first, which takes about 10 minutes). *Leakage:* med. Keep the matching strictly out-of-fold. 4. **Stage-2 boundary model.** Train a second model only on OOF p in [0.2, 0.8]: - Use the schedule features plus the stage-1 OOF p, with stronger regularisation. - Optimise logloss there. - Apply it only to the test rows that fall in that band. - *Why:* Accuracy is decided entirely in the uncertain band, and the global model wastes capacity on easy rows. - *Gain:* +0.04–0.10. *Leakage:* low, provided stage-1 uses nested OOF. 5. **Label-noise cleaning.** Down-weight (do not drop) train rows that are confidently wrong out of fold, e.g. |y − p| > 0.9 across 3 seeds, then retrain. - *Why:* If the ceiling is feature noise or mislabels, removing contradictory anchors sharpens the boundary. - *Gain:* +0.02–0.08. *Leakage:* low. Validate on uncleaned folds. 6. **Prior shift by vintage (EM / Saerens) before thresholding.** Estimate test default prevalence per age bucket from the test predictions themselves, then adjust the posteriors and per-bucket thresholds. - *Why:* Default rate depends strongly on loan age. If the test vintage mix differs, fixed per-grade thresholds are miscalibrated. - *Gain:* +0.02–0.08. *Leakage:* med (uses unlabelled test only, which is generally legitimate). 7. **Segment specialists.** Train separate models for 36 vs 60 term crossed with age bucket (pre-first-payment / active / past maturity), blended with the global model. - *Why:* The schedule arithmetic and the meaning of the lag features differ by segment, so one tree ensemble splits these inefficiently. - *Gain:* +0.02–0.06. *Leakage:* low. 8. **Diverse ensemble with a rank/logit blend.** Combine: - CatBoost with ordered target stats on subgrade×state×purpose, - XGBoost hist, - LightGBM with different feature subsets, - 5 seeds each. Blend in logit space and tune the threshold on blended OOF. - *Why:* This gives a cheap, reliable variance reduction and fits in the remaining 3 days on CPU. - *Gain:* +0.03–0.06. *Leakage:* low. ## Process note On 93k rows, single-run CV noise is about ±0.03–0.05. Your +0.035 and −0.02 results are within that noise. Screen with 3×5 repeated CV using paired fold comparisons before believing any delta below 0.05.
1. **Exact Monthly Installment (EMI) & Expected vs. Actual Principal** - **Idea:** Calculate standard EMI using $P[r(1+r)^n]/[(1+r)^n-1]$, then compute expected remaining principal at month $k$ to subtract from `total_current_balance`. - **Why it works:** You reverse-engineered interest well, but principal paydown is deterministic. Discrepancies between expected and actual balance perfectly flag hidden fees, penalties, or exact arrears amounts. - **Expected gain:** +0.2% - **Leakage risk:** Low 2. **Custom Differentiable Accuracy or Focal Loss Objective** - **Idea:** Replace standard Logloss (BCE) with Focal Loss or a smoothed step-function approximation of Accuracy. - **Why it works:** BCE wastes tree splits pushing confident 0s closer to 0. Accuracy only cares about the decision boundary. Focal loss shifts model capacity strictly to borderline cases. - **Expected gain:** +0.15% - **Leakage risk:** Low 3. **True Post-Loan Debt-to-Income (New DTI)** - **Idea:** Calculate `EMI / (annual_income / 12)` and add it to the existing `debt_to_income` (or check the ratio of EMI to remaining monthly free cash). - **Why it works:** Bank `debt_to_income` is often calculated *pre-loan*. The post-loan DTI reveals borrowers pushed over the absolute mathematical threshold of bankruptcy. - **Expected gain:** +0.1% - **Leakage risk:** Low 4. **Tree Algorithm Diversification (CatBoost Blending)** - **Idea:** Train a CatBoost model on your exact v6 features and simple-average the probabilities with LightGBM before applying thresholds. - **Why it works:** Since Accuracy is highly sensitive to the decision boundary, averaging different tree-splitting logic (especially CatBoost's handling of `state_code` and `subgrade`) inherently reduces variance at the threshold. - **Expected gain:** +0.1% - **Leakage risk:** Low 5. **Subgrade-Relative Pricing Premium (Micro-Risk Margin)** - **Idea:** Create a feature: `interest_rate - mean(interest_rate | subgrade)`. - **Why it works:** Underwriters assign discrete grades but use continuous pricing based on hidden risk factors (e.g., credit pulls not in the dataset). This isolates the exact hidden risk premium charged to the borrower. - **Expected gain:** +0.05% - **Leakage risk:** Low 6. **Deterministic Hard Overrides (Post-processing)** - **Idea:** Force predicted probability to 1.0 (or class 1) if `weeks_behind` > X or `missed_payment_count` > Y, overriding the LightGBM output. - **Why it works:** Tree models smooth over extreme deterministic accounting rules to minimize global loss; hard rules perfectly capture the guaranteed-default tail. - **Expected gain:** +0.05% - **Leakage risk:** Low 7. **Nested Target Encoding on High-Cardinality Interactions** - **Idea:** Strict out-of-fold target encoding on cross-features like `state_code` + `home_ownership` + `job_experience`. - **Why it works:** Decision trees struggle to efficiently map 3D categorical interactions. This creates a dense representation of local economic resilience (e.g., "NY renters with 10+ years exp"). - **Expected gain:** +0.05% - **Leakage risk:** Medium (must use strict nested CV to avoid overfitting) 8. **Covariate Shift Correction via Adversarial Validation** - **Idea:** Train a model to predict Train (0) vs Test (1). Drop or penalize the features with the highest importance in this adversarial model. - **Why it works:** Because `ID` is time-ordered, macro-economic drift is guaranteed. Features that drift between train and test ruin decision boundaries for Accuracy on the test set. - **Expected gain:** +0.05% - **Leakage risk:** Low
| Split seed (quick screen) | v5 features | v6 features | Gain |
|---|---|---|---|
| 2026 | 94.391% | 94.515% | +0.12 |
| 42 | 94.366% | 94.493% | +0.13 |
| 7 | 94.42% | 94.481% | +0.06 |
New signals: loan age read from the lender's price list (each sub-grade and rate pair belongs to a narrow issue window), position in the repayment schedule, and comparison with loans of the same grade and age. Per-grade cut-offs (A–B 0.56, C–E 0.53, F–G 0.49) edged out a single cut-off and a stacking model. The gain is real but small and did not reach +0.1 on every split, so v5 stays recommended. v6 is a sensible extra upload if the platform keeps your best score.
Same three models, updated summary (no data rows). Quick screen on v5 features; v5 baseline 94.391%.
| Idea | Suggested by | Screen | Result |
|---|---|---|---|
| Pricing-vintage age from (sub-grade, interest rate) pairs + rate vs sub-grade trend | claude-opus-5-5, gemini-3.1-pro-preview | 94.478% | Kept |
| Schedule position: balance left, months left, near-maturity flag | gpt-6-astra, claude-opus-5-5 | 94.434% | Kept |
| Cohort-relative payment gap (same grade / term / age) | gpt-6-astra | 94.412% | Kept |
| Loan age re-estimated from neighbours' implied payments | gpt-6-astra | 94.41% | Kept |
| Smooth (isotonic) loan-age curve | gpt-6-astra, claude-opus-5-5 | 94.391% | No gain, dropped |
| Amortisation fit quality (integer-payment residuals, rate / term perturbation) | gpt-6-astra, claude-opus-5-5 | 94.393% | No gain, dropped |
| Other monthly debt implied by debt-to-income | gemini-3.1-pro-preview | 94.384% | No gain, dropped |
| Pseudo-labelling confident test loans | gemini-3.1-pro-preview | 94.466% | No gain, dropped |
openai/gpt-6-astra — full advice (Round 2)
1. **Recover discrete origination cohorts instead of a rolling-age proxy.** Fit robust piecewise-constant upper quantiles of `last_week_pay` versus ID, separating 3-/5-year terms; use cohort age, age uncertainty, and cohort-relative arrears. **Why:** rolling p90 can smear vintage boundaries or underestimate age when many neighbours stopped paying. **Expected gain:** +0.10–0.35 accuracy points. **Leakage:** low without labels; check whether train/test feature pooling is permitted. 2. **Model integer payments and partial-payment residuals.** Compute scheduled cumulative interest at `floor(k)` and `ceil(k)`; add distance to integer `k`, fractional interest remaining, and residuals under alternative weekly-to-monthly calendar alignments. **Why:** v5’s continuous inversion can blur missed, partial, or extra instalments; these distinctions should matter near the boundary. **Expected gain:** +0.08–0.25 points. **Leakage:** low. 3. **Add explicit maturity and early-settlement diagnostics.** Calculate full-term scheduled interest, `interest_receive / full_term_interest`, months until maturity, and implied outstanding principal after `k` payments; flag large apparent arrears despite near-complete repayment. **Why:** an old loan with no recent EMI may be completed rather than defaulted. Keep implied loan principal distinct from `total_current_balance`, which may cover other accounts. **Expected gain:** +0.05–0.20 points. **Leakage:** low. 4. **Train regularized experts for distinct repayment regimes.** Fit specialists for F–G, likely matured loans, and large interest/week disagreement; blend each with the global model using strongly shrunk weights. **Why:** the same payment-history discrepancy can mean different things across risk and lifecycle segments. Select routing and weights using nested OOF predictions. **Expected gain:** +0.05–0.18 points. **Leakage:** medium if routing or weights are repeatedly tuned on the same validation set. 5. **Engineer contractual affordability and credit-utilization features.** Add amortized EMI/monthly income, principal/annual income, revolving balance/limit, unused revolving credit/monthly income, and EMI relative to DTI-implied monthly debt service; handle zero denominators explicitly. **Why:** reconstructs repayment capacity with nonlinear ratios that trees may learn inefficiently, especially for distressed F–G borrowers. **Expected gain:** +0.03–0.12 points. **Leakage:** low. 6. **Optimize a small, shrunk family of accuracy thresholds.** Compare the global `.52` with thresholds by grade group and repayment lifecycle, shrinking segment thresholds toward the global optimum. **Why:** pooled model scores may be differently calibrated across the exact segments producing residual errors. Require gains across folds, not just pooled OOF accuracy. **Expected gain:** +0.02–0.10 points. **Leakage:** medium; threshold-search overfitting is the main risk. 7. **Extract within-vintage pricing residuals.** Estimate typical `interest_rate` conditional on subgrade, term, and inferred ID cohort; add rate residual, within-cohort rate rank, and distances to common rate tiers. **Why:** pricing deviations may reveal underwriting information or finer origination timing beyond the recorded grade. **Expected gain:** +0.01–0.08 points. **Leakage:** low with label-free estimation. 8. **Try conservative, agreement-gated pseudo-labeling.** Retrain one diverse learner using only test examples where the model families agree at extreme probabilities, with soft targets and low sample weights; retain the original ensemble as an anchor. **Why:** 40k additional feature vectors could stabilize sparsely populated cohort/grade combinations. Validate by hiding labels in an inner-fold simulation first. **Expected gain:** +0.00–0.08 points; degradation is plausible. **Leakage:** medium if validation labels influence pseudo-label generation or selection.
anthropic/claude-opus-5-5 — full advice (Round 2)
# Ranked untried ideas (v5 → v6)
1. **Recover exact issue vintage from the (subgrade, interest_rate) pricing table**
- *Why:* Lending-Club-style lenders repriced each subgrade in discrete steps. A given (subgrade, rate) pair maps to a narrow issue window. Cross that with ID order to get a much sharper loan age than rolling p90, which tightens both "weeks behind" and "k − age". Near-threshold errors are mostly age-noise.
- *Build:* For each subgrade, segment ID by rate changes, then compute per-row age = snapshot − segment start, interpolated by ID.
- *Expected gain:* +0.2 to 0.5
- *Leakage risk:* Low. Uses train+test features only, no target.
2. **Amortisation fit-quality features**
- *Why:* A clean payer's interest_receive matches a schedule point almost exactly. Missed, partial or late payments and prepayments leave a residual. That residual separates "behind but curing" from "charged off" in F-G.
- *Features:*
- frac(k) and interest_receive − interest(round(k)).
- Interval width of k when the rate is perturbed by ±0.005 to account for rounding.
- k computed with 5yr vs 3yr terms, to catch mislabeled term.
- *Expected gain:* +0.15 to 0.4
- *Leakage risk:* Low.
3. **Prepayment vs default disambiguation**
- *Why:* A low interest_receive relative to age means either default or principal prepaid early, since less balance means less interest. These two cases look identical to current features.
- *Features:*
- Ratio interest_receive / sched_interest(age).
- Implied remaining principal if all scheduled payments were made, divided by loan_amnt.
- last_week_pay ≥ (age − small) together with an interest shortfall, which suggests prepayment.
- *Expected gain:* +0.1 to 0.3
- *Leakage risk:* Low.
4. **Specialist second-stage model on the uncertain band** (OOF p in roughly [0.3, 0.75])
- *Why:* Errors concentrate there. A model trained only on hard rows, with stronger regularisation and features 1–3, avoids being dominated by easy rows. Fit it on OOF predictions, using the base p as a feature, with nested folds.
- *Expected gain:* +0.05 to 0.2
- *Leakage risk:* Medium. You must use strict nested OOF or CV will inflate.
5. **Segment-wise decision thresholds**
- *Why:* With an accuracy metric, the optimal cut differs by base rate. F-G have a much higher prior than A-B, so one global 0.52 is suboptimal.
- *How:* Optimise the threshold per grade-group × term, or per gap bucket, on OOF. Keep at most 4–6 segments and shrink toward 0.52.
- *Expected gain:* +0.05 to 0.15
- *Leakage risk:* Medium. Overfitting risk, so validate by repeated CV.
6. **Deterministic rule audit / post-processing**
- *Why:* Some segments are near-pure on OOF, yet the blend still errs there from noise:
- k ≥ term·12 − 1, or full scheduled interest received → 0.
- lwp = 0 with age ≫ 0 → 1.
- weeks behind > X → 1.
- *How:* Hard-override segments with ≥99.5% OOF purity.
- *Expected gain:* +0.03 to 0.1
- *Leakage risk:* Low.
7. **Pseudo-label confident test rows** (p < 0.05 or > 0.95), then retrain
- *Why:* 40k test rows, many of them in the same ID/vintage windows as the hard cases. This also helps the neighbour/age features calibrate.
- *How:* One round, test rows weighted 0.5.
- *Expected gain:* +0.03 to 0.1
- *Leakage risk:* Low to medium. Keep pseudo-labels out of validation folds.
8. **Replace fixed .4/.1/.5 blend with an OOF-optimised logistic stack plus 3–5 seed bags per model**
- *Why:* Seed variance near the threshold is likely ~0.05pt. A stacker can also learn per-segment model trust, for example CatBoost better on F-G. This is cheap on CPU if you reuse params.
- *Expected gain:* +0.03 to 0.08
- *Leakage risk:* Low.
**Suggested order for 3 days:**
- **Day 1:** ideas 1 and 2.
- **Day 2:** ideas 3, 6 and 8.
- **Day 3:** ideas 4, 5 and 7, then final submit. Choose one conservative and one aggressive submission.google/gemini-3.1-pro-preview — full advice (Round 2)
1. **Exact EMI & Reverse-Engineered Debt**: Compute exact monthly installment (EMI) using `loan_amnt`, `interest_rate`, and `loan_term` via standard amortization formula, then calculate `pre_existing_debt = (debt_to_income * annual_income / 12) - EMI`. * *Why:* Exposes the borrower's hidden non-loan monthly financial burden; explicitly models Lending Club-style backend logic. * *Expected gain:* +0.15 to +0.25 points. * *Leakage risk:* Low. 2. **Grade/Subgrade-Specific Thresholds**: Instead of a global 0.52 threshold, optimize thresholds locally per `loan_grade` (e.g., A might need 0.55, G might need 0.45) to maximize accuracy on OOF, using scipy.optimize. * *Why:* Directly attacks your "remaining errors in F-G" issue. Accuracy optimal thresholds shift drastically based on subgroup base-rate and predicted probability densities. * *Expected gain:* +0.10 to +0.20 points. * *Leakage risk:* Low. 3. **Confident Pseudo-Labeling**: Take test rows where your ensemble predicts <0.02 or >0.98, assign them as hard labels (0 and 1), concatenate to train, and retrain the models. * *Why:* At 94%+ accuracy, your extreme predictions are highly reliable. Feeding them back helps tree algorithms crystallize the boundaries for the messy middle (threshold) cases. * *Expected gain:* +0.10 to +0.15 points. * *Leakage risk:* Low (transductive learning is standard/allowed unless explicitly banned). 4. **Credit Utilization Ratios**: Calculate `revolving_balance / total_revolving_limit` and `total_current_balance / annual_income`. * *Why:* Classic, overwhelmingly strong credit risk predictors. If you haven't explicitly coded these divisions, trees spend too many splits approximating them. * *Expected gain:* +0.10 points. * *Leakage risk:* Low. 5. **Interest Rate Pricing Anomalies**: Calculate `interest_rate - rolling_median(interest_rate)` over an ID-sorted window of 3000, partitioned strictly by `loan_subgrade`. * *Why:* Lenders assign base rates by subgrade, but penalize (bump the rate) for hidden risk factors not in the dataset. This isolates that "hidden risk premium." * *Expected gain:* +0.05 to +0.10 points. * *Leakage risk:* Low. 6. **Focal Loss / Hard-Sample Weighting**: Switch LGBM/XGB objective to Focal Loss, or linearly weight training rows based on OOF error (e.g., weight = 1 + absolute_error_from_previous_model). * *Why:* You noted errors are concentrated "near threshold". This forces the trees to stop splitting easy 0.1/0.9 cases and focus entirely on the 0.4-0.6 boundary. * *Expected gain:* +0.05 to +0.08 points. * *Leakage risk:* Low. 7. **Expected Principal vs Balance Mismatch**: Using the 'k' (implied payments) from your v5 magic, calculate exact principal amortized at month k. Compare this to `total_current_balance`. * *Why:* Exploits your successful v5 magic further. If `total_current_balance` drops slower than mathematical amortization implies, it strictly signals partial payments/fee accumulations (pre-default). * *Expected gain:* +0.05 to +0.15 points. * *Leakage risk:* Medium (depends on whether `total_current_balance` is loan-specific or global). 8. **Expanding Time-Aware Target Encoding**: Sort by ID, and compute an expanding mean of `default` for interactions like `loan_purpose` + `home_ownership` and `state_code` + `loan_grade`. * *Why:* ID is time-ordered. Default base rates drift chronologically due to macroeconomics. Expanding mean prevents leak while capturing exact temporal risk drift better than static grouping. * *Expected gain:* +0.05 points. * *Leakage risk:* Medium (safe if properly shifted by 1 row).
| Split seed | v4 features | v5 features |
|---|---|---|
| 2026 | 93.2% | 94.4% |
| 42 | 93.15% | 94.38% |
| 7 | 93.21% | 94.41% |
The new signal: work backwards from the interest each borrower has paid to how many monthly payments that covers, then compare it with the loan's age. All three AI advisors proposed it independently. It uses only the loan's own details, never the answer, and the gain held on every split (+1.20, +1.23, +1.20 pts). Blend: 0.4 LightGBM, 0.1 XGBoost, 0.5 CatBoost, cut-off 0.52. Expect the leaderboard a little below the honest estimate, as with v4.
Three models reviewed a written summary of the approach (no data rows) and proposed untried ideas. They advised only; every prediction comes from the tree models. Quick screen on the same splits; the v4-feature baseline scores 93.2%.
| Idea | Suggested by | Screen | Result |
|---|---|---|---|
| Payments implied by interest received (invert the repayment schedule) | gpt-6-astra, claude-opus-5-5, gemini-3.1-pro-preview | 94.3% | Kept |
| Wider time window (3,201 neighbours) + interest vs neighbours | gpt-6-astra, claude-opus-5-5 | 93.25% | Kept |
| Weeks behind measured within the same grade / term | gpt-6-astra | 93.21% | Kept |
| Default rate of neighbouring loans by ID (computed safely inside each split) | gpt-6-astra, claude-opus-5-5, gemini-3.1-pro-preview | 93.2% | No gain, dropped |
| ID gaps and issue-batch density | claude-opus-5-5 | 93.18% | No gain, dropped |
| Rank of weeks paid among neighbours | gpt-6-astra | 93.15% | No gain, dropped |
openai/gpt-6-astra — full advice
1. **Fit a constrained age frontier, not just rolling p90.** Fit monotone quantile splines/isotonic frontiers against ID at q90/q95/q99, allowing issuance-density changes and term-dependent payment caps; add frontier disagreement as uncertainty. This could correct age estimates where local windows mix cohorts or contain many stopped payers. **Estimated gain:** +0.10–0.30 accuracy points. **Leakage:** low with label-free features; check rules before using test covariates. 2. **Add cross-fitted, ID-local default priors.** Compute kernel-smoothed default rates over nearest 200/1,000/5,000 IDs, globally and within grade/term, with shrinkage. Unlike ordinary categorical target encoding, this captures changing underwriting and cohort-specific default prevalence. Exclude the entire validation fold—not just each row—from label sources. **Estimated gain:** +0.05–0.20 points. **Leakage:** med. 3. **Invert the amortisation curve to estimate payments actually collected.** Numerically solve for the payment count implied by `interest_receive`, principal, APR and term; compare it with age-implied and `last_week_pay`-implied counts. This transforms existing interest residuals into interpretable missed-payment counts and exposes inconsistencies. **Estimated gain:** +0.04–0.15 points. **Leakage:** low. 4. **Add conditional peer-distribution features.** Among nearby IDs matched on term and grade/rate, calculate payment-duration percentile, fraction of peers paid further, and percentile of `interest_receive / loan_amnt`. Full distributions can distinguish genuinely delinquent loans from cohorts whose p90 age estimate is misleading. **Estimated gain:** +0.03–0.12 points. **Leakage:** low; test-feature use subject to rules. 5. **Train a small, shrinkage-heavy specialist for difficult loans.** Fit a shallow CatBoost/LightGBM expert for F–G and another for the baseline’s uncertain region, using cross-fitted baseline probabilities to define that region; blend softly with the global model. These subsets may need different interactions that global accuracy optimisation underweights. **Estimated gain:** +0.03–0.12 points. **Leakage:** med unless routing and evaluation are fully cross-fitted. 6. **Optimise decision policy, not just one global threshold.** Jointly tune ensemble weights and regularised grade/age-band thresholds on OOF predictions, then evaluate that policy on an untouched outer split. Small subgroup calibration differences can flip enough boundary cases to matter at 93% accuracy. **Estimated gain:** +0.02–0.10 points. **Leakage:** med from repeated OOF optimisation. 7. **Bag genuinely different age-feature views.** Train compact models using separate frontier scales, quantiles and feature subsets, then select blends by OOF error complementarity rather than standalone score. Existing seed diversity may not address shared errors caused by the same age proxy. **Estimated gain:** +0.02–0.08 points. **Leakage:** low with honest OOF selection. 8. **Try conservative, low-weight pseudo-labelling.** Add only test loans where all model families strongly agree, initially at 0.1–0.2 training weight; validate by treating an outer fold as unlabeled and hiding its labels throughout. Extra samples may stabilise sparse ID-cohort/subgrade regions, though confirmation bias makes this the least reliable bet. **Estimated gain:** +0.00–0.06 points. **Leakage:** med; verify competition rules.
anthropic/claude-opus-5-5 — full advice
# 8 untried ideas, ranked by expected gain
1. **Invert amortisation to get "payments implied by interest_receive."**
- **Idea:** Solve for n (possibly fractional) such that the cumulative interest over the first n payments of the schedule (loan_amnt, rate, term) equals interest_receive. Then add features n_implied, n_implied − last_week_pay/4.33, n_implied − est_age, and frac(n_implied).
- **Why:** This is a per-loan age and delinquency estimate that doesn't depend on noisy neighbour quantiles. A non-integer residual flags partial or late payments. It should separate the F-G loans that currently "look identical."
- **Gain:** +0.15–0.35
- **Leakage:** low
2. **Out-of-fold neighbour default rate along ID.**
- **Idea:** Compute a rolling mean of `default` over the k train neighbours sorted by ID (k = 50/200/1000), excluding self and computed out-of-fold. For test rows, use all train neighbours.
- **Why:** Default labels depend on the snapshot date relative to issue date. Time-local base rates capture cohort and censoring effects that the age estimate misses.
- **Gain:** +0.1–0.3
- **Leakage:** med. The fold split must be contiguous ID blocks or leave-self-out, or CV will be inflated.
3. **Better age envelope on the combined train+test ID sequence.**
- **Idea:** Replace rolling p90 with a monotone upper envelope, e.g. isotonic regression of the rolling p95/max of last_week_pay against ID, or a LightGBM quantile model with α = 0.9–0.97. Compute it over train and test rows merged.
- **Why:** It gives smoother, less biased age estimates, with less noise at window edges and in sparse ID regions. More IDs means tighter estimates.
- **Gain:** +0.1–0.25
- **Leakage:** low (uses only features, not labels)
4. **ID-gap and batch features.**
- **Idea:** Add ΔID to the previous and next loan, local ID density, and a cluster id from large gaps (issue-day proxy). Add each loan's rank of last_week_pay within its batch.
- **Why:** Loans issued in the same batch share an exact age. A within-batch rank is a much sharper "behind" signal than a fixed-width window.
- **Gain:** +0.05–0.2
- **Leakage:** low
5. **Specialist model for the uncertain band and F-G grades.**
- **Idea:** Train a second-stage GBM only on rows where the OOF probability is in 0.25–0.75 or the grade is F-G. Feed it the base OOF probability plus features 1–4, and use it to override the base prediction in that band.
- **Why:** Accuracy is decided entirely near the threshold. A dedicated model can learn interactions that the global model underweights.
- **Gain:** +0.05–0.15
- **Leakage:** med. Use nested OOF to avoid stacking leakage.
6. **Pseudo-label high-confidence test rows and recompute neighbour features.**
- **Idea:** Take test rows with p < 0.05 or p > 0.95 as labels. This densifies the neighbour default rate from idea 2 and adds training rows. Then retrain once.
- **Why:** Test is 30% of the data and interleaved in ID. Denser labels make the local rates less noisy.
- **Gain:** +0.03–0.12
- **Leakage:** med. Evaluate only on the original OOF.
7. **Borrower and duplicate grouping.**
- **Idea:** Hash (annual_income, state_code, total_acc, total_revolving_limit, job_experience, home_ownership) to find repeat borrowers. Add group size and the OOF group default rate. Also find exact-feature duplicates that carry conflicting labels.
- **Why:** "Wrong loans identical to correct ones" may be the same borrower. Otherwise those rows are true label noise, which you can drop from training.
- **Gain:** +0.02–0.1
- **Leakage:** med (group target rate must be OOF)
8. **Direct accuracy optimisation of the blend.**
- **Idea:** Tune blend weights and per-grade (or per-term) thresholds by maximising OOF accuracy with coarse grids. Bag more seeds (5+) of CatBoost, which carries 0.5 weight, so the threshold is stable.
- **Why:** A global 0.49 threshold is unlikely to be optimal across A–G, where calibration differs. Seed bagging reduces flip noise near the boundary.
- **Gain:** +0.02–0.08
- **Leakage:** low, but overfitting risk. Keep grades coarse, e.g. {A–C, D–E, F–G}.
**Validation note:** The ID-time signal drives everything, so validate ideas 2, 4 and 6 with contiguous-ID-block folds as well as random KFold. If the CV gain collapses under block folds, expect the leaderboard gain to as well.google/gemini-3.1-pro-preview — full advice
1. **Grade-Specific Probability Thresholding** - **Idea:** Optimize the classification threshold separately for each `loan_grade` (e.g., A might need 0.45, while F-G might maximize accuracy at 0.55). - **Why it works:** Accuracy is highly sensitive to class priors. Since errors concentrate in F-G (which have much higher baseline default rates), a global 0.49 threshold forces false positives/negatives in edge distributions; local thresholds fix this. - **Expected Gain:** ~0.15% - **Leakage Risk:** Low 2. **Rolling Default Rate by ID (Macro-trend Proxy)** - **Idea:** Create a lagged rolling mean of the `default` target based on the previous 500/1000/2000 chronologically sorted `ID`s. - **Why it works:** If ID is time, the background default rate captures hidden macro-economic shifts (e.g., a recession hitting) or changes in NBFC underwriting strictness over time. - **Expected Gain:** ~0.12% - **Leakage Risk:** High (Must strictly lag/shift the window to avoid including the current row's target). 3. **Exact Reverse-Amortization Math Features** - **Idea:** Use standard financial PMT/IPMT formulas with `loan_amnt`, `interest_rate`, `loan_term`, and `last_week_pay` to calculate the *exact* expected `interest_receive` and remaining principal to the penny, rather than approximations. - **Why it works:** NBFCs use rigid amortization schedules. The exact delta between mathematical expected interest and actual `interest_receive` will perfectly isolate late fees, partial payments, or restructuring. - **Expected Gain:** ~0.10% - **Leakage Risk:** Low 4. **Test-Set Pseudo-Labeling for Rolling Features** - **Idea:** Predict on the test set, take the top/bottom 5% most confident predictions, append them to the train set, and recalculate all rolling ID features. - **Why it works:** The rolling 90th-percentile "age" feature likely breaks down or degrades at the chronological boundary between train and test sets; interleaving test rows stabilizes these time-windows. - **Expected Gain:** ~0.08% - **Leakage Risk:** Medium (Pseudo-labeling can amplify model bias if confidence is poorly calibrated). 5. **Chronological Cross-Validation (Time-Series Split)** - **Idea:** Abandon random K-Fold and use an ID-sorted block split (train on past IDs, validate on future IDs) to retune Optuna hyperparameters. - **Why it works:** The CV (93.30%) > LB (93.06%) gap indicates time-leakage in random folds. Trees tuned on random folds overfit to the future; tuning on a chronological split will select hyperparameters that actually generalize to the test set. - **Expected Gain:** ~0.06% - **Leakage Risk:** Low 6. **Re-train Tabular MLP / Stacking on MVP Features** - **Idea:** Re-run the Tabular MLP and Stacking pipelines *specifically including* the newly discovered "weeks behind" and expected interest features. - **Why it works:** You discarded these methods before the ID insight. Tree models (LGBM/XGB) make correlated errors on borderline instances; an MLP trained on the magic features will behave differently on the "identical-looking" loans, providing massive ensemble diversity. - **Expected Gain:** ~0.05% - **Leakage Risk:** Low 7. **K-Nearest Neighbors (KNN) Target Mean (CPU Faiss)** - **Idea:** Extract only the 10 most critical numerical features, run KNN (k=5, 15), and append the mean target of those neighbors as a new feature. - **Why it works:** You noted "many wrong loans look identical to correct ones." Trees struggle to draw boundaries through dense, overlapping clusters; a distance-based feature helps the tree probabilistically weigh the local neighborhood. - **Expected Gain:** ~0.04% - **Leakage Risk:** Medium (Must be calculated strictly out-of-fold during CV). 8. **Borrower Free Cash Flow Proxy** - **Idea:** Calculate: `(annual_income / 52) * (1 - debt_to_income/100) - EMI`. - **Why it works:** DTI is a ratio, but absolute weekly liquidity determines if a borrower defaults after missing a single paycheck. This quantifies their exact cash buffer in dollars. - **Expected Gain:** ~0.03% - **Leakage Risk:** Low
| v4 model | Accuracy (0.5) |
|---|---|
| LightGBM (seed 1) | 93.19% |
| LightGBM (seed 2) | 93.21% |
| XGBoost | 93.13% |
| CatBoost | 93.24% |
Loan IDs were issued in time order, so comparing each loan's weeks paid with neighbouring loans of the same age shows how many weeks behind the borrower is. That single insight lifted every model above 93%. The four models are blended (0.3 LightGBM, 0.2 XGBoost, 0.5 CatBoost) with a cut-off of 0.49. Our independent re-run matched the claimed 93.30% exactly, and our file agrees with the original on 99.73% of loans.
| v3 model | Accuracy (0.5) | Tuned | Honest estimate | AUC |
|---|---|---|---|---|
| Stacked (logistic meta-model) | 91.85% | 91.87% | 91.83% | 0.9645 |
| Stacked (LightGBM meta-model) | 91.84% | 91.85% | 91.83% | 0.9646 |
| Two-level stack (logistic + LightGBM meta) | 91.86% | 91.87% | 91.83% | 0.9648 |
| LightGBM v3 · 255 leaves, deep | 91.77% | 91.83% | 91.81% | 0.9637 |
| Hill-climb weighted blend | 91.77% | 91.83% | 91.81% | 0.9638 |
| LightGBM v3 · 127 leaves | 91.71% | 91.75% | 91.71% | 0.9632 |
| CatBoost v3 | 91.6% | 91.71% | 91.69% | 0.9617 |
| XGBoost v3 | 91.51% | 91.6% | 91.6% | 0.9606 |
| ExtraTrees v3 | 91.13% | 91.33% | 91.31% | 0.9583 |
| Neural network (tabular MLP) | 90.16% | 90.17% | 90.12% | 0.9397 |
The big step came from rebuilding each loan's repayment schedule and comparing it with the interest actually received. Test loans look statistically identical to training loans (check score 0.4996, where 0.5 means no difference), so no re-weighting was needed. Both LightGBM models were repeated with seeds 7, 42, 2026, and every run scored above 91.6%. The remaining errors sit mostly in grades F–G (about 13–14% wrong vs 6% in grade A): these loans look alike but end differently, which is why 100% is not reachable.
| v2 model | Accuracy (0.5) | Tuned | Honest estimate | AUC |
|---|---|---|---|---|
| Stacked (logistic meta-model) | 91.07% | 91.15% | 91.11% | 0.9545 |
| CatBoost v2 | 90.86% | 91.04% | 90.98% | 0.9524 |
| Simple average | 90.81% | 91.01% | 90.96% | 0.9529 |
| LightGBM v2 · 127 leaves | 90.81% | 90.95% | 90.91% | 0.9522 |
| LightGBM v2 · 63 leaves | 90.72% | 90.93% | 90.9% | 0.9518 |
| XGBoost v2 | 90.37% | 90.59% | 90.56% | 0.9472 |
What changed: interest received ÷ expected to date; share of total lifetime interest received; weeks left on loan; balance ÷ income; revolving balance ÷ loan; rate × repayment progress; peer comparisons within sub-grade, state and purpose (12); category frequencies (3); CatBoost added; the four models combined by a stacking model. LightGBM repeated with seeds 7, 42, 2026 — all scored above 90.6% at the default cut-off, so the gain is stable. 100% is not achievable: some loans look identical in the data but end differently.
v5 is the recommended upload (leaderboard 0.9428, rank 1). v6 is an optional extra upload with a small, unconfirmed gain. v4 (0.9306) is the safe fallback. v3 is the next backup and the v1 files are the last resort; the v1 backup agrees on 98.85% of loans. Leaderboard accuracy is only known after upload.
Repayment progress and interest received vs expected are among the strongest signals. Flag loans whose received interest falls behind schedule within the first few months.
In this data grade A loans default most (27.4%) and grade E least (18.6%). Combine grade with repayment behaviour before tightening approval.
3-year loans default at 27.4% versus 15.3% for 5-year loans. Monitor short-tenure loans closely.
Re-score active loans monthly and route high-probability accounts to collections before they miss a payment.