Data Science — Model Testing

A/B testing process

1. Hypothesis: Define expected outcome
2. Sample size: Calculate required users
3. Randomization: Assign groups
4. Analysis: Compare results

Sample size calculation

from statsmodels.stats.power import NormalIndPower

power_analysis = NormalIndPower()
sample_size = power_analysis.solve_power(
    effect_size=0.2,
    alpha=0.05,
    power=0.8
)
print(f"Required sample size: {sample_size:.0f}")

Statistical test

from scipy import stats

# Two-sample t-test
t_stat, p_value = stats.ttest_ind(control_group, treatment_group)

if p_value < 0.05:
    print("Statistically significant")
else:
    print("Not significant")

Results

def analyze_ab_test(control, treatment):
    control_mean = np.mean(control)
    treatment_mean = np.mean(treatment)
    lift = (treatment_mean - control_mean) / control_mean * 100
    
    return {
        'control_mean': control_mean,
        'treatment_mean': treatment_mean,
        'lift': lift
    }

Best practices

1. Run test long enough
2. Don't peek at results early
3. Consider novelty effects
4. Segment results

Mini Practice

1. Calculate sample size
2. Run A/B test
3. Analyze results
4. Make data-driven decisions

Up Next

Continue with Big Data - Large-scale data processing.