# Train/test splits & data leakage

Code With Purpose · English · Free starter lesson

## Concept

Supervised learning uses features X and known targets y to learn a mapping. Classification predicts categories; regression predicts numeric values. Training data is used to fit a model. Held-out data helps estimate performance on unseen examples.

Leakage occurs when information unavailable at prediction time reaches training or preprocessing. Fitting a scaler or imputer on the entire dataset before a split leaks information from held-out data. A Pipeline fits preprocessing on the training data with the model.

Random splits suit some independent observations, but time series often need chronological splits and repeated entities may need grouped splits. Use validation data or cross-validation for model choices and reserve the final test set for evaluation. This small demo requires scikit-learn; its synthetic score is not a real-world performance claim.

## Example

```python
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=200, n_features=6, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=7)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
print(len(X_train), len(X_test))
print(round(model.score(X_test, y_test), 3))
```

## Expected output

```text
150 50
The next line is test accuracy; the value can vary by library version.
```

## Exercise

A churn dataset includes cancellation_date recorded after cancellation. Can you use it to predict future churn? Explain and remove it from candidate features.

## Suggested solution

```
# No: the date is unavailable before the outcome and leaks target information.
# If df is your real dataset:
# X = df.drop(columns=["churn", "cancellation_date"])
# y = df["churn"]
# Audit every other feature's availability at prediction time too.
```

## Takeaway

Only use information you would have when making the prediction.

Reference: https://scikit-learn.org/stable/common_pitfalls.html

https://codewithpurposeai.in/lesson-ml-splits.html?lang=english