Startseite
Podcasts
Behind the Craft Podcast
How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel

How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel

Behind the Craft vor 1 Monat

0:00

53:52

Copy the AI system I use to run a one-person $1M+ business:
https://www.behindthecraft.com

Subscribe to my practical AI newsletter read by 150K+
professionals: https://creatoreconomy.so

Shreya and Hamel teach the industry standard course for AI
evaluations taken by 4,500+ students from OpenAI, Google, and
more. In our interview, I asked them to give live feedback on the
AI evals that I built for my skills and demo how anyone can run
evals with ChatGPT/Claude and their free evals skill. Don’t miss
this up to date primer on evals from the real experts.

P.S., I asked Shreya and Hamel to give my audience 25% off the
next cohort of their AI evals course. It has 4.7 stars with 900
reviews which is almost unheard of on Maven. Sign up via the link
here:
https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG

We talked about:

(00:00) Why AI evals are completely different now with the latest
models

(02:30) Demo: Live audit of the evals that I built for my AI
skills

(05:14) Top down vs. bottom up evals and why AI sucks at the
latter

(08:16) Spinning up AI agents to grade each eval criteria

(14:32) Demo: Using Shreyas AI skill to run evals in Claude

(23:46) How to review output labels to identify failures

(34:37) How to turn failures into reusable evals

(43:19) Where human judgement is still needed

Resources mentioned:

Free eval skills from Shreya:
https://github.com/ai-evals-course/evals-skills

Do automated evals work?:
https://parlance-labs.com/blog/posts/auto-evals/

Thanks to our sponsors:

Wispr Flow — 4x faster than typing with your voice —
https://ref.wisprflow.ai/peteryang

Get the takeaways:
https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel

Where to find Shreya and Hamel:

Shreya: https://www.sh-reya.com/

Hamel: https://hamel.dev/

Subscribe to this channel - more interviews coming soon!