LLM Evaluation & AI Quality Engineering: Measure, Regression-Test, Release, and Monitor LLM, RAG, and Agent Systems with Python (QA Testing)

Most teams "evaluate" an LLM application by eyeballing a few outputs or trusting one dashboard score. Then a prompt tweak, a model upgrade or a retrieval change quietly breaks production. LLM Evaluation & AI Quality Engineering shows you a better way: AI quality is an engineering lifecycle, not a single evaluation score. You will learn to Define, Measure, Evaluate, Diagnose, Improve, Regression Test, Release, Monitor and Learn, and to treat every measurement as evidence, not truth.

What you will build and learn

• A vendor-neutral Python evaluation package you own, backed by a free public companion repository with 119 automated tests
• Quality objectives, golden datasets, error analysis and failure taxonomies
• Deterministic code evaluators, metrics and rubrics, human annotation with Cohen's kappa and Krippendorff's alpha
• LLM-as-a-judge evaluators you have actually validated, with TPR/TNR meta-evaluation
• Statistics for non-deterministic systems: Wilson confidence intervals, bootstrap, McNemar paired comparison, sample size, pass@k and pass^k
• RAG evaluation: recall@k, precision@k, MRR, nDCG, groundedness, citation validity and retrieval-vs-generation attribution
• AI agent evaluation: sandboxed outcome checks, trajectory and tool-use evaluation, reliability
• Safety and red teaming: prompt injection, canaries, refusal and over-refusal, mapped to the OWASP Top 10 for LLM Applications 2025
• Evaluation-driven development, regression suites for model migration, and cost, latency and reliability trade-offs
• AI quality gates in CI/CD with GitHub Actions, and production observability with OpenTelemetry GenAI semantic conventions, drift detection and triage

Who this book is for

QA engineers, SDETs and test leads moving into AI quality; AI/ML engineers who own LLM, RAG or agent features; and technical leads who sign off AI releases. You need intermediate Python, basic pytest and Git.

Why this book is different

Written by a software QA professional, it turns evaluation results into release decisions you can defend. DeepEval, Ragas, Inspect AI, Arize Phoenix, MLflow, Langfuse, LangSmith and Promptfoo are compared fairly, but you are never locked into a vendor. The final part shows how to run an AI quality engineering programme: RACI, release sign-off, a maturity model, and a practical mapping to NIST AI RMF and ISO/IEC 42001.

23 chapters, 5 parts, 21 figures, 24 tables and 10 appendices. Part of the author's AI Quality Engineering series. Pair it with AI Testing with Python for foundations and RAG in Production for building the systems you will evaluate.