Eight full-time weeks from "engineer who ships LLM features" to "the person a team calls when they can't tell whether their AI product is getting better." Built around one loop: look at the data, measure what you saw, improve, repeat. Eight weeks gets you to a first scoped engagement you can deliver credibly; the broader expertise comes from running many of them.
Time8 wks · ~40 h/wk
Starting pointSWE, ships LLM features
DestinationDeliver a scoped diagnostic engagement
First clientsStartups, enterprise
The thesis
Evals are error analysis with a measurement attached
Almost every failed eval effort fails the same way: a team buys a tool, wires up "hallucination score" and "relevance score", watches a dashboard nobody trusts, and goes back to vibes. The expertise you're building is the opposite order — read a hundred real traces, name the ways they fail, then build a measurement for those failures and prove the measurement agrees with a human.
Everything in this curriculum serves the three moves you named for the consultancy:
Explain
Make a skeptical CTO or a procurement lead understand why "we tried it and it seemed fine" is not a quality process, in five minutes, without jargon.
Do
Walk into a codebase with a chatbot, RAG pipeline or agent in production and, in 2–4 weeks, leave behind a failure taxonomy, a labeled dataset, validated judges, CI gates, and at least one fix you've shown to work.
Teach
Install a weekly ritual and enough skill that the team keeps doing error analysis after you're gone. The handoff is the product; the suite is the artifact.
Each week below ends with a consultant lens — what that week's skill becomes inside a client engagement — and exit criteria you can tick off. Don't move on until you can tick them honestly.
Cadence
How to run the eight weeks
You already know how to call an API and ship a feature, so this is not a reading course. Roughly 30% reading and 70% building, with every week producing an artifact you could show a client. The weekly shape:
Mon–Thu AMRead the week's material. Take notes as if you'll teach it Friday — because you will.Mon–Thu PMBuild the week's project on your capstone system (below). Every week extends the same system; nothing gets built from scratch after week 1.Fri AMWrite. A 1–2 page memo of what you found, in the voice you'd use with a client. Half of these become blog posts in week 8.Fri PMTick exit criteria. Anything unticked spills into the weekend or the next week's Monday — not later.
The capstone. Choose it on day one. If you've shipped an LLM feature with production traces you can keep working on, use that. Otherwise spend the first two days building a small support agent with retrieval over a public help center and a fake orders/refunds API — it is RAG, tool use and multi-turn in one system, so every later week has somewhere to land. Week 4 adds whichever architecture your capstone lacks as an extension, not a second product. A fresh demo has no production traffic, so weeks 2 and 6 each give a substitute for it: recruited-user and simulated sessions, clearly labeled as such, which exercise the whole pipeline without pretending to estimate production quality.
Readings are marked required or optional. Do the required ones before you build; treat optional ones as weekend reading. Each build block names its deliverable, and the deliverable rubrics say what a finished one contains.
Three standing habits: keep a failure journal (every odd LLM output you see, anywhere, with a one-line note on why it's wrong); keep a claims file — every quantitative claim you make gets its n and interval written next to it (week 5 explains why; start now); and from week 3, court a pilot partner — one message a week to a founder or engineering lead whose product has an LLM feature and no eval process. Week 8 needs a partner already lined up.
Curriculum
Eight weeks, eight artifacts
Weeks are strictly sequential: each one's project consumes the previous one's output. If you have only six weeks, compress 7 into 6 and give 8 three days — don't shorten 1–3.
Week 1read 35% · build 65%
Error analysis: the loop that everything else serves
Learn to look at outputs the way a domain expert does, and turn what you see into a counted taxonomy of failure modes.
Learn
Vocabulary that clients confuse: unit test vs. offline eval vs. online eval vs. monitoring vs. benchmark vs. red-team. Draw the diagram; you'll draw it in every discovery call.
The analyze → measure → improve lifecycle. Evals aren't a phase; they're the spec. A fix without a test is a regression waiting for the next model upgrade.
Open coding and axial coding. Read 100 traces, write a free-text note on each ("cited the wrong policy section", "asked a clarifying question it already had the answer to"), then cluster notes into 5–10 failure modes and count them. Stop sampling when new traces stop producing new categories (theoretical saturation).
Start with binary pass/fail plus a written critique. Unanchored 1–5 scales tend to be inconsistent across annotators and don't tell you what to fix; a critique does. It's a default, not a law: anchored ordinal rubrics fit when the client genuinely needs graded severity, and pairwise preferences fit when the decision is "which of these two is better."
Who labels: one principal domain expert ("benevolent dictator") over a committee. Consensus labeling is slow and averages away the expert's judgment.
Pick the LLM feature you've shipped that you trust least. Export 100 production traces (or generate 100 realistic inputs and run them). Open-code every one in a spreadsheet with exactly three columns — trace_id, pass, critique — then axial-code into a taxonomy with counts. Write the Failure Taxonomy Memo: one page, the table below, and the three fixes you'd make first.
Trace
Verdict
Critique (open code)
Failure mode (axial)
t-0417
fail
Quoted the 2023 refund window (30 days) — current policy is 14. Retrieved chunk was stale.
Stale retrieval
t-0418
pass
—
—
t-0419
fail
Asked user for order number; it was in the first message.
Ignores provided context
t-0420
fail
Correct answer, then "I hope this helps! 😊" — brand voice guide forbids emoji.
Tone / style violation
The whole discipline in four rows. Your taxonomy memo is the first artifact you'll show a prospective client.
Consultant lens
This week is the diagnostic sprint you'll sell. A client who has never seen their own failures counted by category has never done an eval, whatever their dashboard says. Practice narrating the memo out loud in five minutes.
Exit criteria
Week 2read 25% · build 75%
Data: traces, datasets, and the annotation tool
Build the data pipeline that makes error analysis repeatable: what to log, how to sample, how to generate what production hasn't given you yet, and how to label it fast.
Learn
Instrumentation. A trace must capture the input, every retrieved chunk, every tool call with arguments and results, the final output, model/prompt version, latency and cost. If a client can't reproduce a failure from a trace, the trace is incomplete.
Sampling. Random for baseline rates; stratified by feature, user segment, or intent for coverage; error-triggered (thumbs-down, retries, escalations) for density. Know which one you're using and say so: error-triggered sampling finds failures faster but produces a sample that says nothing about production rates, so keep a random slice alongside it.
Synthetic data that isn't garbage. Generate from explicit dimensions — persona × scenario × difficulty × edge condition — and have the LLM fill the tuple, not "write 200 questions." Review a sample by hand; LLMs generate suspiciously easy inputs by default.
Coverage sets and production samples answer different questions. A random production sample estimates how often users hit a failure. A synthetic grid or a set of flagged failures exposes failures the sample hasn't hit yet, and its pass rate estimates nothing about production. Tag every example with its origin (production sample, error-triggered, synthetic), report the groups separately, and say which population each number describes. Their average is not a metric.
Three datasets with three jobs. A dev set you iterate prompts and judges on freely. A regression suite that runs on every PR and grows from production failures — developers will adapt to it, and that's fine, because its job is to catch known failures coming back. A small assessment set you run only at milestones and never tune against, so it keeps estimating something. Judges are validated on a holdout carved from the working examples, never on the assessment set: inspecting judge disagreements there would show you the system's failures on it. Running the assessment set rarely helps but doesn't make it independent if its results steer your next change, so treat every look as spending some of its value and refresh it with new examples after a few milestones. The moment a set gives feedback on every PR it has become a dev set; this is the same reason model selection needs a validation split separate from the final test split (scikit-learn's guidance lays it out). Version all three and retire examples when the product changes.
Annotation tooling. Build a custom one; it takes a day and doubles labeling throughput because it shows the trace the way this product needs. Keyboard shortcuts, one screen per trace, pass/fail + critique, nothing else.
Agreement. Cohen's κ between two independent labelers. You-vs-you a week apart measures self-consistency, a weaker check — say which one you ran. A low κ is a prompt to read the disagreements, not a diagnosis: it can come from ambiguous guidelines, a genuinely hard task, or a skewed label distribution (κ is corrected for chance agreement, so with 95% passes a few disagreements on the rare class can drive κ low despite high raw agreement — though annotators who agree on everything still score 1; see the definition). Report κ with the class balance and the disagreement examples next to it.
Read
required Hamel Husain — the sections on building a custom data viewer/annotation tool in the Field Guide; his argument that off-the-shelf viewers lose you the context that matters.
optional A current guide to OpenTelemetry-style tracing for LLM apps — the Langfuse and Arize Phoenix docs both explain the span model well.
Build
Ship a one-page annotation app (Streamlit, FastHTML, or a Next.js page — whatever you're fastest in) that reads traces from a JSONL file and writes labels back. Assemble a 200-example dataset for your capstone: ~100 from a random production sample, ~100 synthetic from a dimensions grid, all labeled with critique and each tagged with its origin. No production data? Recruit 5–10 people (colleagues, a founder community, friends who fit the persona) to use the demo for real tasks of their own choosing while it records full sessions, and add LLM-simulated sessions from the week-4 style persona × goal grid. Tag them recruited and simulated. They give you real-ish traces to code, a taxonomy, and a working pipeline; they say nothing about production rates, and every memo you write on them must say so in the first paragraph. Split by purpose, not just by row: dev, regression, and a sealed assessment set — for example 40 examples sealed, 160 to work with. Report real and synthetic results separately from day one — the real sample estimates production, the synthetic grid finds failures the sample hasn't hit, and their average estimates nothing. Write a one-page labeling guideline. Measure κ against a second labeler (a colleague if at all possible; yourself after a week only as a fallback), read every disagreement, and revise the guideline until you'd trust a judge trained on these labels. Around 0.7 is a reasonable bar for a balanced set; lower can be acceptable when one class is rare, and you should be able to say why.
Consultant lens
Clients will want to skip this and go straight to judges. Don't let them. The dataset and the guideline are what the team keeps; the annotation tool is what makes the weekly ritual survivable. Bring your own tool and adapt it in a day — that's a differentiator.
Exit criteria
Week 3read 35% · build 65%
Evaluators: code, references, and LLM-as-judge done properly
Turn each failure mode into an automated check, and prove each check agrees with the human labels before you trust it.
Learn
The evaluator hierarchy — cheapest first. Deterministic assertions (JSON schema, regex, forbidden strings, length, latency, cost) → reference-based (exact match, field-level F1, embedding similarity) → LLM judge. Many failure modes ("used emoji", "cited a doc not in the retrieved set") need no LLM at all.
Why classic text metrics mostly fail for generation. BLEU/ROUGE reward surface overlap; embedding similarity can't tell "refund is 14 days" from "refund is 30 days." Know them so you can explain why you're not using them.
Judge design. One judge per failure mode, usually a binary output with a reasoning/critique field (pairwise or anchored ordinal when that's the decision being made). Few-shot with real labeled examples from week 2 — both passes and fails. Use a different model (or at least a different prompt lineage) from the system under test to reduce self-preference bias.
Known judge biases: position bias in pairwise comparisons, verbosity bias, self-preference, style-over-substance, and rubric anchoring. Randomize order; test on adversarial pairs.
Judge validation is non-negotiable. Split your labeled data; report the judge's true-positive rate (catches real failures) and true-negative rate (leaves passes alone) against the human on the held-out split — not accuracy, which hides asymmetric errors. Iterate the judge prompt on the dev split only. Report both rates with their class counts and a Wilson interval, and read every disagreement: the pattern in the misses tells you whether the prompt or the guideline is wrong.
The Wilson interval, by hand. For k successes in n with p̂ = k/n and z = 1.96: centre = (p̂ + z²/2n) / (1 + z²/n), half-width = z·√(p̂(1−p̂)/n + z²/4n²) / (1 + z²/n). For the matrix's TPR, 39/43: p̂ = 0.907, z²/n = 0.089, centre = 0.874, half-width = 0.090, interval 78.4–96.3%. Use it instead of the normal approximation whenever n is small or p̂ is near 0 or 1, which describes every judge validation you'll do; the definition and statsmodels' proportion_confint(method="wilson") are the references. Week 5 makes the rest of the statistics rigorous; the interval habit starts now.
Acceptance criteria come from consequences, not a universal threshold. Before you see the matrix, decide what a missed failure costs versus a false alarm for this failure mode. A judge gating refunds needs a far higher true-positive rate than one flagging emoji, and a rule like "90% on both" ignores that. It also ignores uncertainty: with 43 failures in the held-out split, an observed 90.7% is compatible with a true rate anywhere from roughly 78% to 96%. Write the criteria down first, then compare. A perfectly good outcome is "this failure mode stays with a human reviewer and the judge only triages."
Correcting the pass rate for judge error — worked. Once you have sensitivity (TPR) and specificity (TNR), the judge's observed fail rate f̂ is biased; the standard correction is true fail rate ≈ (f̂ + TNR − 1) / (TPR + TNR − 1). Say the judge from the matrix above flags 52 of 200 eval examples, so f̂ = 0.260, with TPR = 0.907 and TNR = 0.922: corrected fail rate = (0.260 + 0.922 − 1) / (0.907 + 0.922 − 1) = 0.182 / 0.829 = 0.220, so the corrected pass rate is 78.0% against a raw 74.0% — the judge's false alarms outnumbered its misses here. For the interval, resample the 43 fails and 77 passes of the validation split (giving a new TPR and TNR) and the 200 eval verdicts together, recompute, repeat a few thousand times, and take the 2.5th and 97.5th percentiles: about 68.6–88.0% for the corrected pass rate, versus a Wilson 67.5–79.6% on the raw rate. It assumes the judge's error rates on the eval set match those on the validation split — same distribution, same mix of failures — which is exactly why the validation split has to resemble what you run the judge on. Two failure modes of the formula: the denominator TPR + TNR − 1 is how much better than a coin flip the judge is, and as it approaches zero the correction blows up — don't correct, don't use the judge. An estimate outside 0–1 means the assumption failed (the eval set's failure mix differs from the validation split) or the counts are too small; clip it, report raw and corrected side by side, and say which. With this estimator the corrected interval is usually wider than the raw one because it adds the validation split's sampling variance, and with 43 fails that addition is large — but compute both rather than assert it, since the comparison depends on the estimator and interval method.
Human: pass
Human: fail
Judge: pass
71agree
4missed failure
Judge: fail
6false alarm
39agree
Judge vs. human on a 120-trace held-out split. TPR = 39/43 = 90.7%, 95% Wilson interval 78.4–96.3%. TNR = 71/77 = 92.2%, interval 84.0–96.4%. Read the intervals: these counts don't establish that either rate is above 90%, and 43 failures is a thin basis for a judge that gates anything consequential. This table with its intervals, per judge, is the artifact that lets a client trust the numbers that follow.
required Evan Miller — Adding Error Bars to Evals, sections 1–3 only, for the framing: why every eval number needs an interval, and where clustering and variance reduction come in. Note that the paper uses normal-approximation intervals, which are fine at benchmark scale but not for a 43-count judge rate — use the Wilson calculation above. The rest of the paper is week 5.
optional Anthropic docs — building evals and the Console's evaluation tool; the guidance on grading with rubrics.
Build
For your top three failure modes: one deterministic check if possible, otherwise an LLM judge. Before you look at any matrix, write the acceptance criteria per failure mode — the cost of a missed failure versus a false alarm, and the minimum TPR and TNR you need, interval included, for the decision this judge will drive. For every judge, produce the agreement matrix above on held-out data with counts and intervals, iterate on dev data, and write a Judge Validation Report: prompt, examples, matrix with intervals, disagreement analysis, corrected pass-rate estimate with its assumptions and interval, and a verdict — automate, triage-only, or human review. Be explicit about which labels do which job: split the ~160 working examples into a judge-dev half you iterate prompts on and a judge-holdout half of ~80 you score judge versions on (the matrix above is an illustrative example from a larger 120-example holdout; with 80 yours will have wider intervals). Scoring each version once on the same holdout is still tuning if each result shapes the next version — and it will. There is no safe number of looks. Use the holdout to iterate, and when the judge is final, label a fresh slice of examples the judge has never been scored on and produce the report's matrix from that; the report states how many versions were scored on the iteration holdout and that the final matrix came from fresh labels. The 40 sealed assessment examples stay closed all week — you'll run them for the first time in week 5, with judges frozen, after the fix. Then run the whole suite against the 160 working examples and compare to the week-1 hand counts, separately for the production sample and the synthetic grid.
Consultant lens
This is where most consultancies are sloppy and where you'll win: never hand a client an unvalidated judge. "Here is how often my judge agrees with your expert, with the interval, and here is the pass rate corrected for its errors" is the sentence that makes you credible to an engineering lead. So is "this failure mode needs a human; the judge can only triage it" — saying it is credibility, not failure.
Exit criteria
Week 4read 30% · build 70%
Task-specific evals: RAG, agents, and multi-turn conversations
Learn the evaluation shape of the three architectures nearly every client runs, and extend your capstone until it exercises all three.
Learn
RAG — evaluate retrieval and generation separately. Retrieval: build ground truth by generating questions from known chunks, then measure recall@k, precision@k, MRR, NDCG. Generation: faithfulness/groundedness to retrieved context, answer relevance, citation correctness. Retrieval failures can produce what looks like hallucination — the model answers faithfully from the wrong chunk — and the split is what tells you whether that's what is happening in this system, so run it before you opine.
Agents — trajectory and final state. Did it call the right tool with the right arguments (per-step correctness)? Did the environment end in the right state (outcome check)? Did it get there without doing anything it wasn't allowed to do (constraint check on consequential actions — a refund agent that reaches the requested state by issuing an unauthorized refund has failed, whatever the final state says)? Outcome and constraints together are the grade; per-step correctness is the diagnosis. Build a transition failure matrix: rows = state it was in, columns = state it went to, cells = how often that transition failed. It shows you where in the loop the agent breaks. Learn pass@k vs. pass^k and why reliability, not peak capability, is what users feel.
Multi-turn — two complementary views. Replay tests that reproduce the conversation up to the first failing turn are your debugging unit: cheap, localized to one turn, and with the context fixed the only thing varying is the model's new response — which still varies, so run each replay several times and report a rate, not a verdict. They don't tell you whether the conversation ultimately succeeded, recovered, or caused new problems downstream, so also grade full sessions end to end: was the user's goal met, did the agent recover from its own errors, did it take any prohibited action along the way. Use an LLM user-simulator with a persona and a goal for coverage, but validate that the simulator behaves like real users before trusting its pass rates.
Other shapes you'll meet: structured extraction (field-level precision/recall), classification (a plain confusion matrix — don't over-think it), summarization (faithfulness + coverage of key points from a reference list).
Read
required Anthropic engineering — Demystifying evals for AI agents (properly this time). Note how it combines transcript grading with outcome grading.
required Hamel Husain / Shreya Shankar — the material on multi-turn eval and transition failure matrices in their evals course FAQ.
optionalRagas documentation — for the metric definitions (faithfulness, context precision/recall), even if you don't adopt the library.
optionalτ-bench (Sierra) and SWE-bench Verified — as examples of outcome-based agent grading with state checks.
Build
Extend the capstone; don't start over. If your capstone is the support agent, its retrieval step is your RAG test bed and its tool calls are your agent test bed — one system, two evaluation shapes. If your capstone is a shipped feature of one shape, add the other as a small extension (a retrieval step over a public help center, or one tool against a fake API), not a second product. Reuse the week-2 dataset and week-3 evaluators. Produce the retrieval-vs-generation split, the transition failure matrix, an outcome check plus a constraint check on consequential actions, and a multi-turn suite with both first-failing-turn replays and end-to-end session grades. A user simulator is required only if you have no real multi-turn traces to work from; otherwise it's optional this week.
Consultant lens
Discovery question #1 becomes "which of the three shapes are you?" and you'll have a ready-made eval design for each. The retrieval/generation split alone usually pays for the engagement — teams routinely tune prompts for a week when the index was the problem.
Exit criteria
Week 5read 45% · build 55%
Statistics, and the first demonstrated improvement
Know when a difference is real, how many examples you need, and how to say so in a sentence a non-statistician trusts — then prove it by fixing something and measuring the fix.
Learn
Non-determinism. Run each example several times; report mean and spread. A single run of a 50-example set can swing by 10 points on its own.
Confidence intervals for a pass rate (Wilson or bootstrap, not the normal approximation for small n). Learn the rough sizes by heart: at n=100, ±~10 points; at n=400, ±~5. That's what "how big should the eval set be?" actually means.
Paired comparisons. Evaluate two prompts/models on the same questions and compare per-question differences; the variance drops dramatically versus comparing two independent means. Bootstrap the paired difference.
Clustered questions (several questions from one document) inflate confidence; cluster the bootstrap by document.
Multiple comparisons, overfitting to the eval set, and contamination. If you iterated on the dev set 40 times, its score is no longer an estimate of anything. Keep the assessment set sealed and run it only at milestones — this week's improvement report is the first — with judges frozen before you open it; the regression suite that runs in CI is a different object with a different job (week 2). Every time an assessment result changes what you do next, it has leaked a little; after a few milestones, replace it. Assume public benchmarks are in the training data.
Two sources of uncertainty, not one. The sample of examples and the judge's own error both widen the interval on a corrected pass rate. Bootstrap them together — week 3's correction, done properly.
Leading vs. lagging metrics. Your judge pass-rate is leading; the client's escalation rate or conversion is lagging. Learn to show the correlation, or admit you can't yet.
optional Narayanan & Kapoor — Evaluating LLMs is a minefield, and their writing on contamination and benchmark validity.
optional A short applied-stats refresher on bootstrap and paired tests — any you like.
Build
Implement the fix. Take the top failure mode from your week-1 taxonomy and implement the fix you proposed there. Evaluate it against the baseline on fresh examples you haven't looked at while designing the fix, five runs each, paired. Write the Improvement Report in lab-report form: hypothesis, n, intervals per failure mode, what got better, what regressed (a fix for one failure mode routinely worsens another), what it cost in latency and tokens, and what you'd still need to find out. This is the assignment the whole course has been pointing at — analyze → measure → improve, with the improve step actually demonstrated. optional Write the statistics as a small library of your own (Wilson interval, clustered bootstrap, paired-difference bootstrap, sample-size calculator). It's a good way to learn them; for client work, reuse a well-tested implementation such as scipy or statsmodels rather than shipping your own.
Consultant lens
Enterprise buyers in particular will have someone in the room who knows statistics. "The new prompt is 4 points better, ±6, so we can't tell yet — here's how many more examples we'd need" is the answer that keeps you in the room. It's also how you tell a client not to ship a change they're excited about. The improvement report is also your first case study: a diagnosed failure, a fix, and honest numbers.
Exit criteria
Week 6read 25% · build 75%
Infrastructure: evals in CI, tooling, and production monitoring
Make the suite run without you: on every prompt change, and continuously on a sample of production traffic.
Learn
The tooling landscape, by shape not by brand. CLI/CI-first (promptfoo), agent-and-research-first (Inspect, from the UK AI Security Institute), hosted platforms with tracing + datasets + judges (Braintrust, LangSmith, W&B Weave), open-source observability-first (Langfuse, Arize Phoenix), metric libraries (Ragas, DeepEval). Go deep on two — one open-source, one hosted — and know the rest well enough to say when they fit.
Evals in CI. A pull request that touches a prompt, a retrieval setting, or a model version runs the regression suite and blocks on regression beyond the interval from week 5. Not the sealed assessment set: once a set gives feedback on every PR, developers optimize toward it and it stops estimating anything. Run the assessment set on its own trigger at milestones — before a release, after a model upgrade, monthly. Budget assertions for cost and p95 latency too. Handle flakiness with repeated runs, not retries-until-green.
Online evals. Sample production traces continuously, run the validated judges, chart pass-rate by failure mode over time. Alert on drift. Route judge-flagged failures into the annotation queue — that's how the regression suite grows — alongside a small random slice, because a dataset built only from flagged traces over-represents whatever the judges already know how to catch.
Guardrails vs. evals. Guardrails run at request time and must be fast and conservative; evals run offline or async and can be slow and thorough. Clients conflate them constantly.
Weak labels from users: thumbs, edits, retries, escalations. Noisy, biased, and still the best free signal you'll get — use them to prioritize what to sample, not as ground truth.
Read
requiredpromptfoo docs — getting started and the CI integration pages; and either the Langfuse or the Phoenix docs on online evaluation.
optionalInspect documentation, if you're going deep on agent evals.
optional Braintrust's and LangSmith's guides on datasets + experiments, mostly to learn the vocabulary clients will already be using.
optional Anthropic engineering — posts on production monitoring and on prompt caching / cost as an eval dimension.
Build
Put the capstone under a GitHub Actions workflow: on PR, run the regression suite five times, post a comment with pass-rate ± interval per failure mode versus main, fail on significant regression. Put the assessment set on a separate, manually triggered workflow that writes its result to the claims file. Add online sampling: a job that pulls 50 recent traces a day, runs judges, and updates a small dashboard; flagged traces land in the annotation tool. If the capstone has no traffic, feed the same job from a nightly batch of simulated sessions plus whatever recruited-user sessions you've gathered that week, labeled as such on the dashboard — the loop's mechanics are what you're building, and you'll swap the source for real traffic on the first engagement. Write the Runbook: how to add a test case, how to re-validate a judge, what to do when the dashboard moves.
Consultant lens
The runbook and the CI gate are the handoff. If the team can't add a test case from a production failure in ten minutes after you leave, the engagement failed. Be tool-agnostic in public: your value is the method, and recommending the client's existing stack when it's adequate builds trust.
Exit criteria
Week 7read 50% · build 50%
Benchmarks, model selection, safety and governance
Read public benchmarks critically, run a defensible model bake-off, red-team your own systems, and speak the enterprise governance dialect.
Learn
Public benchmarks and what they actually measure: MMLU / MMLU-Pro, GPQA, MATH, HumanEval, SWE-bench Verified, ARC-AGI, LiveBench, LMArena, HELM. Learn each one's construction, its known contamination, and why a 3-point gap on any of them says almost nothing about your client's task.
Reading model cards and system cards. What the lab evaluated, what it didn't, and how to translate that into "what should we test ourselves?"
Model-selection bake-off method: your dataset, your validated judges, 3–4 candidates, cost and p95 latency alongside quality, paired comparison with intervals, and a decision memo that names the trade-off.
Red-teaming and adversarial evals. Jailbreaks, prompt injection (critical for tool-using agents — injected text in a retrieved doc or tool result), PII leakage, toxicity, off-policy actions. Automated suites (promptfoo red-team, garak, PyRIT) for coverage; hand-crafted attacks for the specific system.
Governance for enterprise buyers: NIST AI Risk Management Framework, EU AI Act testing and documentation obligations for high-risk systems, ISO/IEC 42001. Learn enough to map your deliverables (taxonomy, judge validation, comparison reports) onto their audit vocabulary.
Read
required A recent system card from Anthropic or OpenAI, read cover to cover with the question "what would I still need to test?"
optionalHELM (Stanford CRFM) — the framework paper, for how to think about multi-metric evaluation.
optionalNIST AI RMF core document (skim) and a summary of EU AI Act obligations for providers and deployers.
Build
Run a bake-off across three models on your capstone and write the decision memo. Red-team the agent test bed — especially prompt injection through tool results — and write a Red-Team Report with reproduction cases that become permanent regression tests. Draft a one-page mapping of your standard deliverables to NIST AI RMF functions for enterprise conversations.
Consultant lens
Two distinct sales conversations open here: startups ask "which model should we use?" and enterprises ask "how do we prove this is safe enough to deploy?" Same method, different memo. Being the person who can answer both is the point of covering this week.
Exit criteria
Week 8read 10% · build 90%
The practice: package it, prove it, run one
Turn seven weeks of artifacts into an offer, and run your first real engagement before the curriculum ends.
Do
Package the engagement using the playbook below: fixed-price diagnostic sprint, scoped build, handoff, optional retainer. Write the two-page service description.
Template every deliverable you've produced: Eval Readiness Assessment, Failure Taxonomy Memo, Labeling Guideline, Judge Validation Report, Improvement Report, Runbook, Red-Team Report, Bake-off Memo. A template is the artifact minus the client.
Publish proof.required one post — the week-5 improvement report is the natural candidate, because it shows a fix that worked with honest numbers. optional a second post from the taxonomy memo, open-sourcing the annotation tool, and a meetup talk titled after your strongest opinion (e.g. "Your judge is lying to you: validate it").
Run one engagement. With the pilot partner you've been courting since week 3, run the diagnostic sprint free or cheap in exchange for a case study, and deliver the taxonomy memo by Friday. If the partner isn't ready, run discovery this week and the sprint the week after: the curriculum ends when the memo is delivered, not on the calendar.
Consultant lens
The case study is worth more than any credential. From here the curriculum becomes a loop: every engagement grows your taxonomy library, your judge prompts, and your templates. Budget a day a week for reading — the field moves — and keep the failure journal forever.
Exit criteria
Explain → Do → Teach
The engagement playbook
Four phases, each with a fixed output. Phases 1–2 are the diagnostic sprint you can sell fixed-price; 3–4 are scoped after the taxonomy exists, because you can't estimate what you haven't seen.
Discovery 1–2 days
What's shipped, what's logged, who the domain expert is, what "good" means in their words, how they decide to ship a prompt change today. Read 20 traces in the room with them.
Output: Eval Readiness Assessment — a one-page scorecard (tracing, dataset, labels, evaluators, CI, monitoring) with a red/amber/green per row.
Error analysis sprint 1 week
Instrument if needed, sample 100–200 traces, stand up the annotation tool, sit with the domain expert and label. Open-code, axial-code, count.
Output: Failure Taxonomy Memo with counts, plus the three highest-leverage fixes. Often the client acts on this alone.
Build 2–3 weeks
Versioned dataset and guideline (dev / regression / assessment), one evaluator per failure mode, judge validation against the expert's labels with acceptance criteria agreed up front, regression suite in CI, online sampling, dashboard — and one fix from the taxonomy shipped with a before/after on fresh examples, so the team has seen the loop close once. Use their stack where it's adequate.
Output: Eval suite in their repo, Judge Validation Reports, one Improvement Report, Runbook.
Handoff 1 week + 30-day check-in
Half-day workshop (below), then pair on the first two weekly error-analysis reviews. Leave when the team adds a test case from a production failure without you.
Output: A team that runs its own weekly review; a check-in at day 30 to catch drift back to vibes.
Explain
The "why evals" talk track
Five arguments, each with a story from your own failure journal. Pick the two that match the room.
Arguments
You can't improve what you can't measure. Every prompt tweak today is a coin flip you're not recording.
Vibes don't scale past one developer. The person who "just knows" when it's good is a bus-factor of one.
Model upgrades regress silently. The next model version will be better on average and worse on three of your failure modes; without a suite you'll learn which from customers.
Evals are the spec. A labeled dataset is the most precise statement of what the product should do that the company has ever written.
Evals compound. Every fix ships with a test; every production failure becomes a test case. Quality ratchets instead of oscillating.
Anti-patterns to name out loud
×A dashboard of generic scores ("hallucination 0.83") that no one can act on.×Unanchored 1–5 ratings with no critique.×An LLM judge that was never checked against a human.×An eval set that was built once and never grown from production.×Buying an eval platform before anyone has read 100 traces.×Reporting a pass rate with no n and no interval.×One "held-out" set that runs on every PR and is also quoted as the unbiased score.×A headline pass rate that averages production traces with synthetic edge cases.
Teach
The half-day handoff workshop
Runs on the client's own traces, never on slides about evals in general. The point is that they do it, with you watching.
0:00 – 0:30Why, using their data. Show three of their own failing traces and the taxonomy counts. Deliver the talk track in their vocabulary.0:30 – 2:00Live error analysis. Everyone labels 15 fresh traces in the annotation tool. Compare labels, argue about disagreements, update the guideline in real time. This is where criteria drift becomes visible and the method clicks.2:00 – 3:00Build one judge together. Pick a failure mode, write the judge prompt from the labeled examples, run it, look at the agreement matrix, fix it once.3:00 – 3:30Install the ritual. Weekly 45-minute review: 30 new traces, update taxonomy counts, add one test case, check the dashboard. Name the owner. Put it on the calendar before you leave the room.
Self-assessment
Am I an expert yet?
"Expert" here is defined by what you can do for a client on a Monday morning, not by what you've read. Eight weeks gets you solidly into the Competent column with one real engagement behind you; the Expert column is what repeated engagements build. Honest markers:
Capability
Competent
Expert
Error analysis
Can produce a taxonomy in a week.
Produces a counted taxonomy in a day from a codebase you've never seen; arrives with hypotheses from the architecture and the discovery call, and revises them from the traces instead of defending them.
Judges
Builds judges that mostly agree with humans.
Sets acceptance criteria from the cost of each error type before seeing the matrix, reports TPR/TNR with intervals and corrected pass rates by default, and says "this one needs a human" when the data says so.
Improvement
Can propose fixes.
Has shipped a fix, shown the before/after on fresh examples with intervals, and named what it regressed.
Statistics
Knows to add intervals.
Tells a client the change they're excited about isn't distinguishable from noise, and how many examples would settle it.
Architectures
Can eval a chatbot.
Has a ready design for RAG, agents and multi-turn, and diagnoses retrieval-vs-generation in the first hour.
Tooling
Knows one platform well.
Recommends the client's existing stack when it's enough, and can say precisely when it isn't.
Teaching
Can explain evals.
Leaves teams that still run the weekly review 90 days later.
Proof
Has done the exercises.
Two public write-ups, one public repo, one case study with a named client.
Review
Deliverable rubrics
Every build block names a deliverable. Write each one for the client's engineering lead, and use the reviewer question as the test: if it can't be answered from the document alone, the deliverable isn't finished. Ask a peer to review against this table before you tick the exit criteria.
Deliverable
Must contain
Reviewer asks
Eval Readiness Assessment playbook · wk 8
One page; a row each for tracing, dataset, labels, evaluators, CI, monitoring; red/amber/green with the one sentence of evidence that justifies each colour; the first thing you'd fix.
Would the client's CTO recognise their own system in it, and could they dispute a colour with evidence?
Failure Taxonomy Memo wk 1
Sampling method and n; 5–10 named failure modes with counts and two example traces each; the three fixes you'd make first and why those.
Could I reproduce the top three counts from the traces? Do the fixes follow from the counts?
Labeling Guideline + dataset wk 2
Pass/fail definition per failure mode with a borderline example each; origin tag on every row; dev / regression / assessment split; κ with class balance and the disagreements that drove revisions.
Could a new annotator label 20 traces from this alone and agree with you?
Judge Validation Report wk 3
Acceptance criteria written before validation; prompt and few-shot examples; matrix with counts and Wilson intervals; disagreement analysis; corrected pass rate with assumptions and interval; verdict (automate / triage / human).
Would I let this judge gate a release, and does the report tell me the consequences if I'm wrong?
Retrieval-vs-generation split wk 4
Retrieval metrics (recall@k, MRR) on question–chunk ground truth; generation metrics (faithfulness, relevance) conditioned on retrieval being correct; the share of end-to-end failures attributable to each stage, with intervals.
If I fix only the index, how much of the failure goes away?
Agent transition matrix + multi-turn suite wk 4
States and transitions named; failure rate per transition with counts; outcome check and constraint check defined; the replay cases and the end-to-end session grades, each with its rate over several runs.
Where in the loop does it break, and can it reach the right end state by a prohibited route?
Improvement Report wk 5
Hypothesis; what changed; fresh-example evaluation, paired, with intervals per failure mode; regressions named; cost and latency delta; open questions.
Did it actually get better, on what, at what cost, and how sure are we?
Runbook wk 6
How to add a test case from a production failure; how to re-validate a judge; what runs on PR versus at milestones; what to do when the dashboard moves; who owns the weekly review.
Could an engineer who wasn't in the engagement add a test case in ten minutes?
Bake-off Memo wk 7
Candidates; your dataset and validated judges; quality with paired intervals, cost, p95 latency on one page; the trade-off named; a recommendation.
Is the recommendation defensible to someone who knows statistics?
Red-Team Report wk 7
Attack categories tried; ≥10 reproducible cases with severity; which are now regression tests; what wasn't covered.
Which of these would have reached production, and are they in CI now?
Sample deliverable: a completed Failure Taxonomy Memo
FAILURE TAXONOMY MEMO · Order-support assistant · 2026-09-19 · 1 page · FICTIONAL — illustrative data; the system, dates, trace IDs and counts are invented to show the expected depth
What we looked at. 100 conversations from 2026-09-08 to 09-14, drawn uniformly at random from the 4,180 sessions that week, so the rates below estimate that week's production, give or take the intervals. One labeler (me), pass/fail plus a written critique per conversation. The support lead independently labeled 30 of them:
Lead: pass
Lead: fail
Me, total
Me: pass
16
1
17 (57%)
Me: fail
3
10
13 (43%)
Lead, total
19 (63%)
11 (37%)
30
Observed agreement 26/30 = 0.867; chance agreement (0.57 × 0.63) + (0.43 × 0.37) = 0.518; κ = (0.867 − 0.518) / (1 − 0.518) = 0.72. The classes are close to balanced, so κ isn't being dragged down by imbalance here. Three of the four disagreements were conversations the lead passed and I failed: a correct answer followed by an unrequested upsell. We decided that counts as a fail and the guideline now says so. The fourth was a tone violation I missed.
Headline. 42 of 100 conversations failed (95% interval 33–52%). Three failure modes account for 33 of the 42 fails, and two of them are fixable without touching the model.
Failure mode
Count /100
95% interval
Example traces
Stale retrieval — answers from a superseded policy document
17
11–26%
t-0417, t-0552
Ignores provided context — asks for an order number or email already in the thread
11
6–19%
t-0419, t-0603
Wrong tool arguments — right tool, wrong order ID or date range
6
3–12%
t-0488, t-0721
Tone / style violation — emoji, exclamation marks, "I hope this helps!"
4
2–10%
t-0420, t-0655
Refuses an in-policy request
3
1–8%
t-0501, t-0699
Other (single occurrences, listed in the appendix)
3
1–8%
—
Counts sum to 44 because two conversations exhibited two failure modes. Tone violations co-occurred with an otherwise correct answer in all four cases; they are cosmetic but the brand guide forbids them, so they stay fails.
The three fixes I'd make first
Re-index with document effective dates and filter to the current version at retrieval time. Addresses the 17 stale-retrieval fails directly; 15 of the 17 cited a policy PDF that has a dated successor in the same folder. This is an index change, not a prompt change. Expected effect: most of a 17-point failure mode, testable on the same 17 conversations plus fresh ones.
Pass the parsed order ID and customer email into the system prompt as structured fields rather than relying on the model to find them in the thread. Addresses the 11 ignores-context fails; in 9 of them the value was in the first user turn.
Add a deterministic check for emoji and exclamation marks as a post-processing guard and as a CI assertion. Cheapest fix on the list, and it makes a whole failure mode a unit test.
I would not start with wrong-tool-arguments: 6 cases split across three distinct causes, and it needs the transition-failure analysis in week 4 before a fix is defensible.
What this memo does not tell you
Whether the failure mix differs by customer segment — the sample wasn't stratified. Next sample should be.
What the escalation or refund-cost impact of each failure mode is. I've asked for the escalation flag to be joined onto the trace table.
Anything about weeks other than 09-08 to 09-14. Policy documents changed on 09-02, which may inflate stale retrieval.
Next step. Fix 1 and fix 3 this week; re-sample 100 fresh conversations after deploy and re-count the same taxonomy, reporting the before/after with intervals. Appendix: the 100 labeled traces with critiques, the labeling guideline v0.2, and the 4 disagreement cases.
That is the depth expected of every deliverable: real counts, intervals, named traces, fixes that follow from the counts, and a section that says what the document cannot claim. Write yours to the same standard before ticking the exit criteria.
Library
Reading list, in the order to read it
Deliberately short, and the required/optional markers per week tell you what can wait. The field moves fast enough that people matter more than papers: follow Hamel Husain, Shreya Shankar, Eugene Yan, Jason Liu, Chip Huyen, Evan Miller, the Anthropic engineering blog, and Narayanan & Kapoor's AI Snake Oil newsletter, and you'll see the next important thing the week it appears.
Hamel Husain & Shreya Shankar's AI Evals for Engineers & PMs course and its public FAQ.If you take one paid course, this is it; the FAQ alone is worth a day.
Landscape
Tooling, by shape
Learn two deeply (suggested: promptfoo + Langfuse, or Inspect + Braintrust) and the rest by shape. Never let a tool choice precede error analysis.
Tool
Shape
Reach for it when
promptfoo
OSS, CLI/CI-first, YAML test cases, built-in red-teaming
Client wants evals in CI this week; red-team coverage.
Inspect (UK AISI)
OSS Python framework, agent- and sandbox-oriented
Agent evals with tool environments; research-grade rigor.
Langfuse
OSS tracing + datasets + scores, self-hostable
Client needs observability and can't send data to a vendor.
Arize Phoenix
OSS tracing + evals, OpenTelemetry-native
Already on OTel; RAG retrieval debugging.
Braintrust
Hosted: tracing, datasets, experiments, judges, CI
Startup that wants the whole loop hosted and fast.
LangSmith
Hosted, tight with LangChain/LangGraph
Client is already in the LangChain ecosystem.
W&B Weave
Hosted, experiment-tracking heritage
ML team already on Weights & Biases.
Ragas / DeepEval
Metric libraries
Borrow metric definitions; rarely the whole system.
Your own
Annotation tool + judge templates + deliverable templates (+ a stats helper if you built one)
Every engagement. This is your leverage.
Progress checkboxes are saved in this browser only. Tool and paper names reflect the landscape as of September 2026; verify current versions before recommending anything to a client.