Gen AI — Evaluation

Why evaluate?

Ensure AI systems produce accurate, helpful, and safe outputs.

Evaluation metrics

Metric	Measures
Accuracy	Correct answers
Relevance	Topic alignment
Coherence	Logical flow
Fluency	Language quality

Automated evaluation

def evaluate_response(response, expected):
    return {
        "exact_match": response == expected,
        "contains_answer": expected.lower() in response.lower(),
        "length_ratio": len(response) / len(expected)
    }

LLM as judge

def llm_judge(question, response):
    prompt = f"""
Rate this response on a scale of 1-5:
Question: {question}
Response: {response}

Criteria: accuracy, completeness, clarity
Output: score and brief explanation
"""
    return client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )

Human evaluation

evaluation_criteria = {
    "accuracy": "Is the information correct?",
    "relevance": "Does it answer the question?",
    "clarity": "Is it easy to understand?",
    "completeness": "Is it thorough?"
}

Benchmark datasets

• MMLU: Multiple-choice knowledge
• HumanEval: Code generation
• TruthfulQA: Factual accuracy
• HellaSwag: Common sense reasoning

Mini Practice

1. Evaluate a model response
2. Build an LLM judge
3. Create evaluation criteria
4. Test with benchmark data

Up Next

Continue with Hallucinations - When AI is wrong.