AI Evaluations Crash Course in 50 Minutes (2025) | Hamel Husain

Today, I want to share a new episode with Hamel Husain.





Hamel has trained 2,000+ PMs and engineers from companies like
OpenAI, Anthropic, and Google on how to run AI evals. In my new
episode, he shares a free master class on how to build evals for
a real AI agent in just 50 minutes using a simple spreadsheet. I
learned a lot from Hamel and I think you will too.





Hamel and I talked about:


(00:00) What the most valuable part of evals is


(01:25) Live walkthrough: Analyzing 100 real production traces


(09:50) Creating the eval criteria using a simple spreadsheet


(24:44) Why binary pass/fail ratings beat 1-5 scores every time


(28:52) The agreement metric trap that fools most PMs


(30:08) True positive and negative rates explained


(36:00) How to set up continuous evals in production





Get the takeaways:
https://creatoreconomy.so/p/ai-evaluations-crash-course-in-50-minutes-hamel-husain





Where to find Hamel:


X: https://x.com/HamelHusain


Website: https://hamel.dev/





Subscribe to this channel – more interviews coming soon!