Model Improvement
What you'll see here
Ten configurations across four levers, judged on validation recall first and F1 second.
The four levers
Depth / capacity
32 → 64-32 → 128-64-32 hidden units
Optimizer
SGD (lr 0.01) vs Adam (lr 0.001)
Regularisation
Dropout 0.2 and 0.3 between hidden layers
Imbalance handling
Class weights (9.01 : 0.53) and random oversampling
Every model uses the same split, same preprocessing, same 30 epochs and batch size 128, and a fixed seed of 42 — so differences come from the lever, not from run-to-run noise.
Model performance summary
| Model | Architecture | Optimizer | Dropout | Balancing | Train recall | Val acc | Val recall | Val F1 |
|---|---|---|---|---|---|---|---|---|
| M1 Baseline SGD | 32 | SGD | - | None | 59.12% | 97.6% | 58.56% | 73.03% |
| M2 SGD deeper | 64-32 | SGD | - | None | 74.66% | 98.58% | 75.23% | 85.42% |
| M3 Adam 2 layers | 64-32 | Adam | - | None | 90.43% | 99.25% | 90.54% | 93.06% |
| M4 Adam 3 layers | 128-64-32 | Adam | - | None | 92.23% | 99.08% | 87.84% | 91.33% |
| M5 Adam + dropout | 128-64-32 | Adam | 0.3 | None | 89.64% | 99.35% | 90.99% | 93.95% |
| M6 SGD + class weights | 64-32 | SGD | - | Class weights | 90.99% | 96.75% | 91.44% | 75.75% |
| M7 Adam + class weights | 128-64-32 | Adam | - | Class weights | 99.1% | 97.9% | 90.54% | 82.72% |
| M8 Adam + dropout + CW Selected | 128-64-32 | Adam | 0.3 | Class weights | 91.89% | 99.05% | 91.89% | 91.48% |
| M9 Adam + oversampling | 128-64-32 | Adam | - | Oversampling | 100% | 98.92% | 88.74% | 90.16% |
| M10 Adam + dropout + OS | 128-64-32 | Adam | 0.2 | Oversampling | 99.44% | 98.65% | 91.44% | 88.26% |
Validation accuracy, recall and F1 across all ten configurations
Depth, optimizer and dropout (M1–M5)
- Switching from SGD to Adam is the single biggest jump: recall rises from 58.6% to 90.5% at the same depth-2 architecture.
- A third hidden layer alone (M4) overfits slightly — training recall 92.2% against validation 87.8%.
- Adding dropout 0.3 to the deeper network (M5) closes that gap and delivers the best F1 of the unweighted models at 93.95%.
Class weights and oversampling (M6–M10)
- Class weights lift recall in every case, but without regularisation they cost precision heavily — M6 drops to 64.7% precision, meaning one in three alarms is false.
- M8 reaches the highest validation recall of the whole set at 91.89% while holding precision above 91%.
- Oversampling reaches similar recall but memorises the duplicated minority rows — training recall of 99.4–100% against 88.7–91.4% on validation.
Imbalance handling compared — same 128-64-32 network
| Strategy | Model | Train recall | Val recall | Val precision | Val F1 | Read |
|---|---|---|---|---|---|---|
| None | M5 | 89.64% | 90.99% | 97.12% | 93.95% | Highest precision, misses most failures |
| Class weights | M8 | 91.89% | 91.89% | 91.07% | 91.48% | Selected: best recall, precision held |
| Oversampling | M10 | 99.44% | 91.44% | 85.29% | 88.26% | Same recall, clear memorisation gap |
AI: Which lever mattered most
Click Generate to produce an evidence-based observation, insight, recommendation, and business benefit for this section.