Train/test splits & data leakage

Understand the concept

Supervised learning uses features X and known targets y to learn a mapping. Classification predicts categories; regression predicts numeric values. Training data is used to fit a model. Held-out data helps estimate performance on unseen examples.

Leakage occurs when information unavailable at prediction time reaches training or preprocessing. Fitting a scaler or imputer on the entire dataset before a split leaks information from held-out data. A Pipeline fits preprocessing on the training data with the model.

Random splits suit some independent observations, but time series often need chronological splits and repeated entities may need grouped splits. Use validation data or cross-validation for model choices and reserve the final test set for evaluation. This small demo requires scikit-learn; its synthetic score is not a real-world performance claim.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=200, n_features=6, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=7)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
print(len(X_train), len(X_test))
print(round(model.score(X_test, y_test), 3))

Output / what to expect

150 50
The next line is test accuracy; the value can vary by library version.

Try it yourself

A churn dataset includes cancellation_date recorded after cancellation. Can you use it to predict future churn? Explain and remove it from candidate features.

# No: the date is unavailable before the outcome and leaks target information.
# If df is your real dataset:
# X = df.drop(columns=["churn", "cancellation_date"])
# y = df["churn"]
# Audit every other feature's availability at prediction time too.

Concept samjho

Supervised learning mein features X aur known targets y se mapping seekhte hain. Classification categories predict karta hai; regression numeric values. Training data se model fit hota hai. Held-out data se unseen examples par performance ka estimate milta hai.

Leakage tab hota hai jab prediction ke time unavailable information training ya preprocessing mein pahunch jaati hai. Split se pehle poore dataset par scaler ya imputer fit karna held-out data ki information leak karta hai. Pipeline preprocessing aur model ko training data par saath fit karta hai.

Random split kuch independent observations ke liye theek hai. Time series ke liye chronological aur repeated entities ke liye grouped split chahiye ho sakta hai. Model choices validation ya cross-validation se karo; final test set evaluation ke liye bachao. Demo mein scikit-learn chahiye. Synthetic score real-world result ka claim nahi hai.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=200, n_features=6, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=7)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
print(len(X_train), len(X_test))
print(round(model.score(X_test, y_test), 3))

Output / kya expect karein

150 50
The next line is test accuracy; the value can vary by library version.

Ab khud try karo

Churn dataset mein cancellation_date cancellation ke baad record hoti hai. Kya future churn predict karne mein isko feature bana sakte ho? Explain karo aur candidate features se hatao.

# No: the date is unavailable before the outcome and leaks target information.
# If df is your real dataset:
# X = df.drop(columns=["churn", "cancellation_date"])
# y = df["churn"]
# Audit every other feature's availability at prediction time too.