Clean a small dataset with Pandas

Understand the concept

A DataFrame is a table-like object with labeled columns. Start with shape, dtypes, and isna().sum() to understand its structure and missing values. Cleaning choices depend on the meaning of the data, not just whether a command removes an error.

drop_duplicates removes duplicate rows by default. That is only appropriate if identical rows are truly accidental copies; repeated transactions can be valid. pd.to_numeric(..., errors="coerce") turns invalid numeric text into missing values, so inspect how many values were affected.

In this tiny teaching example, we fill a missing study-time value with the observed median. This is a demonstration, not a universal rule. In predictive modeling, calculate imputation values from training data only. Install Pandas in your environment before running the example.

import pandas as pd
df = pd.DataFrame({"name": ["Asha", "Ravi", "Asha", "Meera"],
                   "hours": ["2", None, "2", "4"]})
clean = df.drop_duplicates().copy()
clean["hours"] = pd.to_numeric(clean["hours"], errors="coerce")
clean["hours"] = clean["hours"].fillna(clean["hours"].median())
print(clean["hours"].tolist())
print(len(clean))

Output / what to expect

[2.0, 3.0, 4.0]
3

Try it yourself

Before filling missing values, count them. Then create a flag that records which hours values were missing. Why keep the flag?

clean = df.drop_duplicates().copy()
clean["hours"] = pd.to_numeric(clean["hours"], errors="coerce")
clean["hours_was_missing"] = clean["hours"].isna()
print(clean["hours_was_missing"].sum())  # 1
clean["hours"] = clean["hours"].fillna(clean["hours"].median())
# The flag preserves the distinction between observed and filled values.

Concept samjho

DataFrame ek table jaisa object hai jiske columns ke naam hote hain. shape, dtypes aur isna().sum() se shuru karo: kitni rows hain, types kya hain aur missing values kahan hain. Cleaning ka decision data ke meaning se aata hai, sirf error hatane se nahi.

drop_duplicates default mein identical rows hata deta hai. Yeh tabhi sahi hai jab duplicates accidental copies hon; repeated transactions valid bhi ho sakte hain. pd.to_numeric(..., errors="coerce") invalid numeric text ko missing bana deta hai. Dekho kitni values change hui.

Is chhote example mein missing study time ko observed median se fill karenge. Yeh teaching example hai, har dataset ka rule nahi. Predictive modeling mein imputation value sirf training data se nikaalo. Example chalane se pehle environment mein Pandas install karo.

import pandas as pd
df = pd.DataFrame({"name": ["Asha", "Ravi", "Asha", "Meera"],
                   "hours": ["2", None, "2", "4"]})
clean = df.drop_duplicates().copy()
clean["hours"] = pd.to_numeric(clean["hours"], errors="coerce")
clean["hours"] = clean["hours"].fillna(clean["hours"].median())
print(clean["hours"].tolist())
print(len(clean))

Output / kya expect karein

[2.0, 3.0, 4.0]
3

Ab khud try karo

Missing values fill karne se pehle count karo. Ek flag banao jo record kare ki kaunsi hours values missing thi. Yeh flag useful kyun hai?

clean = df.drop_duplicates().copy()
clean["hours"] = pd.to_numeric(clean["hours"], errors="coerce")
clean["hours_was_missing"] = clean["hours"].isna()
print(clean["hours_was_missing"].sum())  # 1
clean["hours"] = clean["hours"].fillna(clean["hours"].median())
# The flag preserves the distinction between observed and filled values.