Data Science — Data Cleaning

Handle missing values

import pandas as pd
import numpy as np

# Drop missing values
df.dropna(inplace=True)

# Fill with mean/median/mode
df.fillna(df.mean(), inplace=True)
df.fillna(df.median(), inplace=True)
df.fillna(df.mode().iloc[0], inplace=True)

# Forward/backward fill
df.fillna(method='ffill', inplace=True)
df.fillna(method='bfill', inplace=True)

Remove duplicates

df.drop_duplicates(inplace=True)

Handle outliers

# IQR method
Q1 = df['column'].quantile(0.25)
Q3 = df['column'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

df = df[(df['column'] >= lower) & (df['column'] <= upper)]

Fix data types

df['date'] = pd.to_datetime(df['date'])
df['category'] = df['category'].astype('category')
df['numeric'] = pd.to_numeric(df['numeric'], errors='coerce')

String cleaning

df['text'] = df['text'].str.lower()
df['text'] = df['text'].str.strip()
df['text'] = df['text'].str.replace(r'[^\w\s]', '', regex=True)

Mini Practice

1. Handle missing values
2. Remove duplicates
3. Detect outliers
4. Fix data types

Up Next

Continue with Time Series - Temporal data analysis.