Home

Preprocessing

What you'll see here

Every transformation applied to the data, and the guarantee that none of them saw the test set.

Training split
16,000
888 failures (5.55%)
Validation split
4,000
222 failures (5.55%)
Hold-out test
5,000
282 failures (5.64%)
Transformation steps
1
Duplicate value check

0 duplicate rows in train, 0 in test — nothing removed

2
Missing value treatment

36 cells in train (V1, V2) and 11 in test, filled with the training median

3
Outlier treatment

None applied — extreme sensor values are retained as failure signal

4
Feature engineering

Not applicable: predictors are already ciphered, transformed sensor signals

5
Encoding

Not required — no categorical variables in the dataset

6
Scaling

StandardScaler fitted on the training split and applied to validation and test

No data leakage
The imputer and scaler are fitted on the training split only and then applied to validation and test. Test.csv is never used for imputation, scaling, weighting, model selection or threshold choice — it is scored exactly once, by the final model.
Data preparation for modeling
  • The 20,000 training rows are split 80/20 with stratify=Target so both sides keep the same 5.55% failure rate.
  • Class weights of 9.009 for failures and 0.529 for healthy units are computed from the training split and used in the weighted models.
  • Random oversampling of the minority class is tested as an alternative to class weights, applied to the training split only.
  • Validation and test keep their natural failure rate — no resampling is applied to either.
Cost assumptions used for impact sizing
EventAssumed unit cost
Inspection (false positive)$5,000
Repair (true positive)$15,000
Replacement (false negative)$60,000

Absolute costs are illustrative and preserve the stated ordering (replacement >> repair > inspection); the percentage saving is what should be read, not the dollar amounts.

AI: Preprocessing & leakage control

Click Generate to produce an evidence-based observation, insight, recommendation, and business benefit for this section.