A layered approach to agent evaluation: from deterministic checks and trajectory evals to LLM-as-judge and human review.

Three things that broke in production and exactly how I fixed them

Why we stopped routing agents one step at a time and compiled the full pipeline upfront

How splitting conversation and understanding into two changed everything

Things that bit me building production agents with LangGraph