Pandas — Performance

Use appropriate dtypes

import pandas as pd

# Instead of object
df['id'] = df['id'].astype('int32')  # Instead of int64
df['category'] = df['category'].astype('category')

Vectorize operations

# Bad
for i in range(len(df)):
    df.loc[i, 'new'] = df.loc[i, 'A'] * 2

# Good
df['new'] = df['A'] * 2

Use built-in methods

# Bad
result = df['A'].apply(lambda x: x.sum())

# Good
result = df['A'].sum()

Chunking

chunks = pd.read_csv('large.csv', chunksize=10000)
results = []
for chunk in chunks:
    results.append(chunk.groupby('col').sum())

Mini Practice

1. Optimize dtypes
2. Vectorize operations
3. Use built-in methods
4. Process large files

Up Next

Continue with Memory - Memory management.