Clean DataFrames: Missing Values, Duplicates and Validation
Turn a messy student register into reliable data. Every example starts with a DataFrame created in Python — no CSV, Excel or database required.
What makes data dirty?
Real tables can contain missing marks, duplicates, inconsistent city spellings, spaces, numbers stored as text, impossible scores and incomplete records. Cleaning means applying explicit rules and recording what changed—not simply removing anything unusual.
| StudentID | Name | City | Marks | Problem |
|---|---|---|---|---|
| 101 | " Asha " | " Varanasi " | "88" | Extra spaces, string score |
| 103 | Meera | VARANASI | None | Missing score, repeated record |
| 104 | " Kabir" | Patna | "108" | Out-of-range mark |
| 105 | Nina | None | "-4" | Missing city, impossible mark |
| 106 | None | Ranchi | "absent" | Name missing, nonnumeric mark |
Run python -m pip install pandas once if needed. We preserve the original data with .copy(deep=True) before changing anything.
Create and inspect deliberately messy data
Start by building your own DataFrame. A snapshot makes the original available for later comparison.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
original = students.copy(deep=True)
print(students)
print("Shape:", students.shape) # (8, 5)
print(students.dtypes)
print("Missing per column:")
print(students.isna().sum())Remember: None is missing; the string "absent" is not automatically treated as missing.
Find empty and missing cells
Use isna() and notna() to search missing values. Blank strings containing spaces also require cleanup.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
print(students.isna().sum())
print(students.loc[students["Marks"].isna(), ["StudentID", "Name", "Marks"]])
print("Missing entries in the table:", int(students.isna().sum().sum()))
print("Known marks:", students["Marks"].notna().sum())Remember: isna() checks missing values; do not compare directly with == None or == pd.NA.
Standardise names and city names
Trim leading/trailing spaces, convert blank strings to missing values and make city names consistent. Nullable string dtype preserves missing values.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
for column in ["Name", "City"]:
students[column] = (
students[column]
.astype("string")
.str.strip()
.replace("", pd.NA)
.str.title()
)
print(students[["Name", "City"]])Remember: Title case is a convenient demonstration; real names may need special-case rules. Do not silently rewrite official identity records.
Detect and remove exact duplicates
Check duplicate rows before removing them. Use the entire row for a safe demonstration; StudentID duplicates are a separate issue.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
print("Repeated exact rows:", int(students.duplicated().sum()))
print(students.loc[students.duplicated(keep=False)])
students = students.drop_duplicates().copy()
print("Rows after removal:", len(students))
print("Repeated IDs:", students["StudentID"].duplicated().sum())Remember: drop_duplicates(subset=["StudentID"]) could discard a genuine conflicting record. Investigate key collisions instead.
Convert numbers stored as text
pd.to_numeric(errors="coerce") parses numeric strings and marks unparseable text, such as "absent", as missing.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
before = students["Marks"].notna()
students["Marks"] = pd.to_numeric(students["Marks"], errors="coerce")
students["Attendance"] = pd.to_numeric(students["Attendance"], errors="coerce")
failed_to_parse = before & students["Marks"].isna()
print("Unparseable marks:", int(failed_to_parse.sum()))
print(students.loc[failed_to_parse, ["Name", "Marks"]])
print(students.dtypes)Remember: Always check how many inputs became missing after conversion. Numeric-looking text might include legitimate formatting that needs explicit treatment.
Validate marks using business rules
Student marks must lie between 0 and 100. Invalid entries must be flagged for investigation. This beginner example sets impossible scores to missing rather than guessing a correction.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
students["Marks"] = pd.to_numeric(students["Marks"], errors="coerce")
invalid = students["Marks"].notna() & ~students["Marks"].between(0, 100)
print("Invalid values found:")
print(students.loc[invalid, ["StudentID", "Marks"]])
students.loc[invalid, "Marks"] = float("nan")
print(students[["Name", "Marks"]])Remember: Never replace a score of 108 with 100 automatically unless a documented policy authorises that choice.
Decide which missing values to fill
Different columns require different decisions: keep unknown marks missing, use "Unknown" for missing city, and optionally fill missing attendance with the observed median.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
students["Marks"] = pd.to_numeric(students["Marks"], errors="coerce")
invalid = students["Marks"].notna() & ~students["Marks"].between(0, 100)
students.loc[invalid, "Marks"] = float("nan")
students["City"] = students["City"].astype("string").str.strip().replace("", pd.NA).fillna("Unknown")
median = students["Attendance"].median()
if pd.notna(median):
students["Attendance"] = students["Attendance"].fillna(median)
students["MarksMissing"] = students["Marks"].isna()
print(students[["Name", "City", "Marks", "MarksMissing", "Attendance"]])Remember: Imputation is a modelling choice. Record it and check whether median filling is appropriate for your problem.
Remove records missing required fields
If a name is essential for the exercise, find incomplete records and remove them only after reviewing. Optional columns should not trigger row deletion automatically.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
students["Name"] = students["Name"].astype("string").str.strip().replace("", pd.NA)
print("Missing names:", students["Name"].isna().sum())
rejected = students.loc[students["Name"].isna()].copy()
students = students.dropna(subset=["Name"]).copy()
print("Rejected for review:\n", rejected[["StudentID", "Name"]])
print("Remaining rows:", len(students))Remember: Use dropna(subset=[...]) instead of dropna() if other fields may legitimately remain missing.
Validate the cleaned DataFrame
Make your expectations executable with assertions. Review records if an identifier appears more than once with conflicting details.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
students["Name"] = students["Name"].astype("string").str.strip().replace("", pd.NA)
students = students.dropna(subset=["Name"]).drop_duplicates().copy()
students["Marks"] = pd.to_numeric(students["Marks"], errors="coerce")
invalid = students["Marks"].notna() & ~students["Marks"].between(0, 100)
students.loc[invalid, "Marks"] = float("nan")
assert students["StudentID"].is_unique, "Conflicting student IDs need review"
assert students["Name"].notna().all()
assert students["Marks"].dropna().between(0, 100).all()
print("All defined checks passed")Remember: Assertions are useful teaching checks; for production pipelines use explicit validation and error handling rather than relying only on assertions.
Identify outliers without deleting them
A rare value can still be real. Use the IQR method as an exploratory flag only after checking the valid score range. This independent illustration adds more observations so you can see a potential outlier.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
# A slightly larger manually-created sample makes outliers visible.
scores = pd.DataFrame({
"Name": ["Asha", "Ravi", "Meera", "Kabir", "Nina", "Rahul", "Tara", "Ishan"],
"Marks": [52, 54, 56, 57, 60, 62, 64, 98]
})
valid = scores.loc[scores["Marks"].between(0, 100), ["Name", "Marks"]]
q1, q3 = valid["Marks"].quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
unusual = (valid["Marks"] < low) | (valid["Marks"] > high)
print("Potential outliers:")
print(valid.loc[unusual, ["Name", "Marks"]]) # Ishan: 98
# Flag for review; do not delete automatically.Remember: A small class register is not suitable for strong outlier conclusions. Flag, investigate, and retain records until you have evidence.
Check how much data changed
An audit describes the number of records entering and leaving your process. It helps identify accidental losses.
import pandas as pd
students = pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
original = students.copy(deep=True)
working = original.copy(deep=True)
working["Name"] = working["Name"].astype("string").str.strip()
working = working.drop_duplicates().dropna(subset=["Name"]).copy()
report = {
"initial_rows": len(original),
"remaining_rows": len(working),
"removed_rows": len(original) - len(working),
"original_missing_cells": int(original.isna().sum().sum())
}
print(report)
assert original.equals(students)Remember: This is a simple audit. The complete project below also records conversions, invalid marks and filled values.
Clean the entire class register with one function
This fully runnable Python program creates eight records directly, works on a copy, reports every cleaning decision, validates the result and outputs six rows. It does not invent missing exam marks.
Open the Python source · View automated tests
"""Pandas 0 to Infinity, Level 3: DataFrame cleaning without external files."""
import pandas as pd
def make_messy_students():
"""Create an intentionally messy but entirely in-memory class register."""
return pd.DataFrame([
{"StudentID": 101, "Name": " Asha ", "City": " Varanasi ", "Marks": "88", "Attendance": 92},
{"StudentID": 102, "Name": "Ravi", "City": "ranchi", "Marks": "42", "Attendance": 75},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 103, "Name": "Meera", "City": "VARANASI", "Marks": None, "Attendance": 80},
{"StudentID": 104, "Name": " Kabir", "City": "Patna", "Marks": "108", "Attendance": 50},
{"StudentID": 105, "Name": "Nina", "City": None, "Marks": "-4", "Attendance": None},
{"StudentID": 106, "Name": None, "City": "Ranchi", "Marks": "absent", "Attendance": 65},
{"StudentID": 107, "Name": "Rahul", "City": "varanasi", "Marks": "91", "Attendance": 95},
])
def clean_students(raw):
"""Return a new cleaned table and an explicit audit; leave raw untouched."""
df = raw.copy(deep=True)
audit = {"input_rows": len(df), "missing_before": df.isna().sum().to_dict()}
for column in ("Name", "City"):
df[column] = df[column].astype("string").str.strip().replace("", pd.NA).str.title()
audit["exact_duplicate_rows_removed"] = int(df.duplicated().sum())
df = df.drop_duplicates().copy()
supplied_marks = df["Marks"].notna()
df["Marks"] = pd.to_numeric(df["Marks"], errors="coerce")
audit["unparseable_marks"] = int((supplied_marks & df["Marks"].isna()).sum())
bad_marks = df["Marks"].notna() & ~df["Marks"].between(0, 100)
audit["marks_outside_0_100"] = int(bad_marks.sum())
df.loc[bad_marks, "Marks"] = float("nan")
df["Attendance"] = pd.to_numeric(df["Attendance"], errors="coerce")
bad_attendance = df["Attendance"].notna() & ~df["Attendance"].between(0, 100)
audit["invalid_attendance"] = int(bad_attendance.sum())
df.loc[bad_attendance, "Attendance"] = float("nan")
audit["rows_without_name_removed"] = int(df["Name"].isna().sum())
df = df.dropna(subset=["Name"]).copy()
audit["cities_marked_unknown"] = int(df["City"].isna().sum())
df["City"] = df["City"].fillna("Unknown")
attendance_median = df["Attendance"].median()
audit["attendance_filled_with_median"] = (
int(df["Attendance"].isna().sum()) if pd.notna(attendance_median) else 0
)
if pd.notna(attendance_median):
df["Attendance"] = df["Attendance"].fillna(attendance_median)
df["MarksMissing"] = df["Marks"].isna()
df = df.reset_index(drop=True)
assert df["StudentID"].is_unique, "Review conflicting StudentID records manually"
assert df["Name"].notna().all(), "Name is required"
assert df["Marks"].dropna().between(0, 100).all(), "Invalid marks remain"
assert df["Attendance"].dropna().between(0, 100).all(), "Invalid attendance remains"
audit["output_rows"] = len(df)
audit["unresolved_marks"] = int(df["MarksMissing"].sum())
return df, audit
if __name__ == "__main__":
original = make_messy_students()
cleaned, report = clean_students(original)
print("Original records:\n", original.to_string(index=False))
print("\nCleaned records:\n", cleaned.to_string(index=False))
print("\nCleaning audit:")
for key, value in report.items():
print(f"- {key}: {value}")
Expected audit highlights
| Check | Result |
|---|---|
| Input rows | 8 |
| Exact duplicate removed | 1 |
| Row with missing required name removed | 1 |
| Impossible marks flagged | 2 |
| Unparseable mark identified | 1 |
| Attendance filled using median | 1 |
| Output rows | 6 |
| Unknown marks retained as missing | 3 |
Practice safely: run python python/pandas/03-clean-dataframes/student_cleaning.py. Run the repository tests with python -m unittest discover -s python/pandas/03-clean-dataframes -p "test_*.py". For real school data, keep a separate reviewed-rejection log and the raw input in secure storage.
Your turn — clean a mini shop table
Construct a DataFrame with Item, Price and Stock. Introduce one repeated row, one blank item, the text "free" in Price, and a negative stock value. Remove exact duplicates, mark unparseable prices as missing, flag negative stock and print an audit.
Show a worked solution
import pandas as pd
shop = pd.DataFrame({
"Item": [" Pen ", "Notebook", "Notebook", "", "Eraser"],
"Price": ["10", "50", "50", "5", "free"],
"Stock": [30, 12, 12, 4, -3]
})
raw = shop.copy(deep=True)
shop["Item"] = shop["Item"].astype("string").str.strip().replace("", pd.NA)
duplicates = int(shop.duplicated().sum())
shop = shop.drop_duplicates().dropna(subset=["Item"]).copy()
shop["Price"] = pd.to_numeric(shop["Price"], errors="coerce")
bad_stock = shop["Stock"] < 0
shop.loc[bad_stock, "Stock"] = pd.NA
print(shop)
print({"raw_rows": len(raw), "duplicates": duplicates,
"remaining_rows": len(shop), "missing_prices": int(shop["Price"].isna().sum()),
"missing_stock": int(shop["Stock"].isna().sum())})
assert raw.shape[0] == 5What you should now understand
Inspect → find missing values → clean text → identify duplicates → convert data types → validate numerical ranges → decide which fields may be filled → preserve an audit → verify the final table.
Official documentation: Working with missing data · to_numeric · drop_duplicates.
Level 4 is available: sorting, grouping, summaries, groupby() and agg().