Pandas — Memory Optimization

Check memory usage

import pandas as pd

df = pd.read_csv('large.csv')
print(df.info(memory_usage='deep'))
print(df.memory_usage(deep=True).sum() / 1024**2, "MB")

Optimize memory

# Reduce numeric types
for col in df.select_dtypes(include=['int']).columns:
    df[col] = pd.to_numeric(df[col], downcast='integer')

for col in df.select_dtypes(include=['float']).columns:
    df[col] = pd.to_numeric(df[col], downcast='float')

# Convert strings to categories
for col in df.select_dtypes(include=['object']).columns:
    if df[col].nunique() < len(df) * 0.5:
        df[col] = df[col].astype('category')

Read in chunks

chunks = pd.read_csv('large.csv', chunksize=100000)
processed = pd.concat([chunk.process() for chunk in chunks])

Mini Practice

1. Check memory usage
2. Optimize dtypes
3. Use categories
4. Process in chunks

Up Next

Continue with Pipes - Method chaining.