# Who Grades the Coach?

### Building an evaluation system that catches an AI tutor’s mistakes before a kid does

The app I work on has an AI coach that gives real-time feedback to kids working through fractions, decimals, and long division. It’s warm, it’s patient, and — like any LLM — it is capable of quietly getting worse in ways that never show up as a bug.

A bad prompt edit doesn’t crash anything. It just starts leaking an answer before the child has worked for it, or inventing a number that isn’t actually in the problem, or teaching the wrong method with total, cheerful confidence. None of that trips a unit test. It ships clean, and it stays there until a parent notices.

So I built the system that’s supposed to notice first.

## The problem with grading an AI tutor

You can’t spot-check your way to confidence here. A regression might show up in one concept out of a dozen, on one phrasing out of a hundred, only some of the time. The only way to catch it is to read what the coach actually says, at scale, against a real teaching standard, on a schedule — and to be honest with yourself about what “better” actually means when the thing doing the measuring is *also* an LLM.

## How the system works

Each test case pairs a real question with a **teaching anchor** — the specific method a kid should come away understanding. Not just “the answer is right,” but *why*: “compare fractions by a common denominator or benchmark,” for example, not “convert to decimals” or any other technically-valid-but-off-target explanation.

The pipeline runs like this:

1. **Probe case** — a real question and student answer, mapped to a concept’s teaching anchor.

2. **Coach** — the actual production prompt generates a reply. Not a mock, not a simplified stand-in — the real thing a kid would see.

3. **Judge** — an independent model, often from a different model family than the coach itself, scores the reply against the anchor.

4. **Safety gate** — three of six criteria are zero-tolerance. A single failure blocks the pipeline, no matter how good the average score looks.

5. **Report** — a timestamped, versioned scorecard, trended against the last run and the last week.

## LLM As Judge: What the Independent Judge actually catches

Up to this point, the judge grading the coach had always come from the same model family as the coach itself — close to a student grading their own homework. Recently I finally got a genuinely different model doing the grading, and the difference showed up immediately.

One example makes the point well. A student was asked to solve 46 × 10, and the coach’s explanation was the classic shortcut: “just add a zero to the end” (46 → 460). That gets the right answer here. But it’s not real understanding — it’s a memorized trick that quietly breaks the moment decimals enter the picture (try “adding a zero” to 4.6 × 10 or 0.5 × 10 and it stops making sense). The actual concept the curriculum wants taught is that multiplying by 10 shifts every digit one place to the left — a rule that works for any number, not just this one.

The same-family judge scored that explanation as fine. The independent judge didn’t, and said exactly why: “the ‘add a zero’ trick rather than the expected place-value anchor of noticing each digit shifts one place left... a common misconception-prone shortcut that doesn’t teach place-value understanding.”

It wasn’t a one-off. The same independent grading caught a hint on 467 + 358 that never mentioned carrying the ten — even though 7 + 8 = 15 requires it — and a long-division hint that walked a student toward the answer without ever checking that a remainder has to be smaller than the divisor, which was the actual point of that question. Three separate cases, all technically-not-wrong, all missing the real teaching point, and all invisible to a judge grading from the same model family as the thing it was grading.

## The rubric: six criteria, and they’re not all equal

A high average score is meaningless if the coach handed a kid the answer even once. So the rubric splits cleanly into two tiers:

The safety-gate criteria don’t get averaged in with the rest. They fail the build, full stop, independent of everything else — because a coach that’s 95% excellent and 5% “here’s the answer” isn’t 95% safe. It’s not safe.

## A weekly loop that’s only allowed to make small moves

1. **Measure.** Run the full probe suite against the live prompt. Every run writes a new, versioned report — nothing overwrites history, so trends are real trends, not vibes.

2. **Target.** Pick the one or two weakest concepts. Never try to fix everything in one pass.

3. **Verify.** Re-run before and after. A score change under the noise floor doesn’t count as an improvement — LLM scoring is not perfectly deterministic, even at temperature 0, and I’d rather under-claim progress than chase noise.

4. **Ship.** Open a pull request with the evidence attached — the before/after reports, the gate results, the reasoning. A person merges it. The system never merges child-facing coaching logic on its own.

## The fix that looked good and wasn’t

Here’s the part I actually want to tell you about, because it’s the whole reason this system is worth having.

I once made a tightly scoped prompt edit to improve two weak concepts. On the next run, both scores went up. If I had looked only at the intended outcomes, I would have called it a success.

But an unrelated concept—one the instruction explicitly said not to affect—fell from 95 to 75.

That was the more important result.

Rather than rationalize the drop as model noise, I ran a control: the same probe suite, with only the prompt edit removed. The concept returned to its previous range. The regression followed the edit.

The lesson was not simply that this particular wording was wrong. It was that prompt scope is not a dependable safety boundary. Once an instruction enters the context, the model may generalize it beyond the concepts it names—even when the prompt explicitly tells it not to.

So the change stopped before review. I reverted it, replaced it with a narrower intervention, and documented the failure mode so the team would not have to rediscover it later.

That is what the evaluation system is really protecting against. It does not only test whether the product is improving. It tests whether the proposed fix is safe enough to become part of the product.

The two intended gains were real. They still were not enough to ship.

## What the system is not allowed to do

A few rules keep this honest:

- **Never grade the grader.** The rubric and its thresholds can’t be loosened to inflate a score. Only a proven bug in the judge itself — isolated, tested, called out explicitly — can touch the grading logic.

- **A regression is not a vibe.** Every claimed improvement is checked against a noise floor across repeated runs, not read off a single before/after diff.

- **The gate ships with the code.** Safety criteria run in CI, not just in a report. A pull request can’t merge over a leaked answer.

- **A person always signs off.** Every coaching-logic change ships as a draft. A human is the last check before a child ever sees it.

---

*This is the eval and safety infrastructure behind an AI math coach for kids — built so that “the AI seems fine” is never the bar, and so the system that watches the coach is at least as disciplined as the coach itself.*