Pandas — Data Cleaning

Detect missing values

import pandas as pd
import numpy as np

df = pd.DataFrame({'A': [1, 2, np.nan], 'B': [np.nan, 2, 3]})

print(df.isnull())
print(df.isnull().sum())

Drop missing values

df.dropna()           # Drop rows with any NaN
df.dropna(subset=['A'])  # Drop rows where A is NaN
df.dropna(thresh=2)   # Drop rows with less than 2 non-NaN

Fill missing values

df.fillna(0)              # Fill with 0
df.fillna(df.mean())      # Fill with mean
df.fillna(method='ffill') # Forward fill
df.fillna(method='bfill') # Backward fill

Interpolate

df.interpolate()

Replace

df.replace({np.nan: 0})

Mini Practice

1. Detect missing values
2. Drop missing rows
3. Fill missing values
4. Use interpolation

Up Next

Continue with Duplicates - Removing duplicates.