# Clean a small dataset with Pandas

Code With Purpose · English · Free starter lesson

## Concept

A DataFrame is a table-like object with labeled columns. Start with shape, dtypes, and isna().sum() to understand its structure and missing values. Cleaning choices depend on the meaning of the data, not just whether a command removes an error.

drop_duplicates removes duplicate rows by default. That is only appropriate if identical rows are truly accidental copies; repeated transactions can be valid. pd.to_numeric(..., errors="coerce") turns invalid numeric text into missing values, so inspect how many values were affected.

In this tiny teaching example, we fill a missing study-time value with the observed median. This is a demonstration, not a universal rule. In predictive modeling, calculate imputation values from training data only. Install Pandas in your environment before running the example.

## Example

```python
import pandas as pd
df = pd.DataFrame({"name": ["Asha", "Ravi", "Asha", "Meera"],
                   "hours": ["2", None, "2", "4"]})
clean = df.drop_duplicates().copy()
clean["hours"] = pd.to_numeric(clean["hours"], errors="coerce")
clean["hours"] = clean["hours"].fillna(clean["hours"].median())
print(clean["hours"].tolist())
print(len(clean))
```

## Expected output

```text
[2.0, 3.0, 4.0]
3
```

## Exercise

Before filling missing values, count them. Then create a flag that records which hours values were missing. Why keep the flag?

## Suggested solution

```
clean = df.drop_duplicates().copy()
clean["hours"] = pd.to_numeric(clean["hours"], errors="coerce")
clean["hours_was_missing"] = clean["hours"].isna()
print(clean["hours_was_missing"].sum())  # 1
clean["hours"] = clean["hours"].fillna(clean["hours"].median())
# The flag preserves the distinction between observed and filled values.
```

## Takeaway

A clean table should preserve meaning and document assumptions.

Reference: https://pandas.pydata.org/docs/getting_started/intro_tutorials/

https://codewithpurposeai.in/lesson-data-cleaning.html?lang=english