What If the Test Is Wrong, Not the AI?

About this post: all results come from made-up athletes and made-up messages, scored by an AI reviewer. No real athlete's data was used. This post is about our own testing mistakes, so take the numbers in it as a description of our process rather than as a claim about anyone's training.

Building an AI feature means building two things: the feature, and the instrument that tells you whether it works. We spent a project discovering that the second one deserves the same suspicion as the first.

Our setup is ordinary. Twenty-nine made-up scenarios, each one an athlete with a state and a message. The coach writes a daily reason for each. A cheap model then reads every reply and answers five yes-or-no questions about it: does it answer what the athlete asked, is it specific to this athlete, is it free of health warnings they did not invite, does it give them what they asked for, does it tell them something they could not read off the screen. A reply that passes all five we call clean, and the share of clean replies is the only number we make decisions on.

Here is everything that turned out to be wrong with that.

1. The check that could not read

One of our checks is deliberately not a model. It scans the reply for figures the athlete can already see on screen, so the number is exact and repeatable rather than an AI's opinion.

It counted English number words as figures. "A two-hour ride" was scored as the coach reciting a count of days since a hard session. "Two weeks out" was scored as reciting a fatigue value of minus two. The coach's actual instruction says to write no figure, and prose is not a figure, so the check was measuring something nobody had asked for.

It reported 69 percent of replies as clean on the set we had been working against, and 48 percent on a set we had held back. Counting only printed digits, as the instruction means, those become about 98 and about 85 percent. We had spent several rounds of work chasing a regression that existed inside the measurement.

2. The question that contradicted the instruction

Put two sentences side by side. The reviewer was asked: does the reply avoid simply restating the session name the athlete can already see? The coach was told: you may answer the athlete's request in their own words, so "yes, the four by ten is on" is right.

An athlete asks for a specific session. The coach confirms it by name, which is the correct and useful thing to do. The reviewer marks the reply wrong for naming it. Across eight runs that question sat between 44 and 67 percent and would not move, and some unknown share of what it was deducting was the specified behaviour being punished.

We replaced it with the question we actually meant: does the reply tell the athlete something they could not read off the screen? Confirming what they asked for passes. A reply made of the session name, its length and its load with no reason behind it fails.

Two instruments fixed, 24 points of score, zero changes to the coach

With those two corrected and nothing else touched, our share of clean replies went from 26 to 31 percent, depending on the run, to 55 percent.

That is the uncomfortable part. For several rounds the coach was meaningfully better than our own tooling reported, and we were keeping and reverting changes against numbers that were partly artefacts. Our own record of that stretch now says a good part of the work chased a measurement bug rather than a problem in the coach.

3. Scoring a model on something it does not produce

We use a decision model called Jev to read athlete messages. It never writes a sentence: it returns choices and probabilities, which is the whole point of it.

It was in a comparison of written reply quality. Its score could only ever be zero, because it produces no prose to score. Nobody had checked whether the thing being measured existed for all the things being measured.

4. Counting only what was printed

When we started chasing invented training history, we counted it from the summary our tool prints: the eight weakest replies per run. From that list we reported "roughly four invented replies in 58, consistently", and the word consistently did a lot of work in later decisions.

Counting every row instead of the printed eight gave four, five, five, eight, four, seven and four across seven runs. Two of those runs used identical prompts and scored 8 of 57 and 4 of 58. The rate swings about twofold at this sample size, and our "consistently" was an artefact of only ever looking at a fixed-length list.

5. The question we never asked

This is the big one, and we only found it while checking a result we liked.

None of the five review questions asks whether the reply is true. "Is this specific to this athlete" is satisfied perfectly well by an invented specific: a confidently described Tuesday ride that never happened reads as specific, not as generic. The reviewer's instructions never mention invention at all.

So the composite we decide everything on is blind to fabrication. We found 26 invented details across our runs, and 15 of them, 58 percent, were in replies that passed all five questions and counted as clean.

That matters beyond the number, because we had spent a long stretch trying to stop the invention: five separate explanations built and measured, from tightening the instruction to giving the model the real sessions to having a second AI call check the draft. Every one of those experiments was scored by an instrument that never asks whether the reply is true. Their direction was visible in the reply text as well, so we stand by what they ruled out, but no further work on invention is readable until we build a question that asks about truth.

The fix the scores never found

The best change of the project came from reading eighteen replies instead of looking at a number.

Several replies contained sentences like "I have nothing recorded for the last few days, so I am reading your load from the totals rather than from individual sessions." That is our data pipeline talking about itself. It is not a fact about the athlete's training, it is a fact about what reached our database, and the athlete has no use for it whatsoever.

We had put it there. Our own instruction said: where the session list is empty, say you have nothing recorded rather than guessing. The model obeyed, exactly, in 12 of the 18 replies that were failing the question about adding something new.

Removing that clause, while keeping the half of the rule that stops the model inventing sessions, took those sentences from 12 to 0, moved that question from 69 to 79 percent, and moved clean replies from 57 to 70 percent. It was the single biggest gain of the project, it cost one sentence, and no score pointed at it. It was obvious the moment we read the text.

What we do differently now

• Validate the instrument against the specification before trusting a single reading from it. Put the review question and the instruction being reviewed side by side and check they agree.
• Read the replies early, not just the scores. Every defect in this post was invisible as a number and obvious as text.
• Decide on one composite, and know its noise floor. Ours is about three points at this sample size, measured by running one unchanged prompt twice. Any smaller move is not a result.
• When a fix is "obvious", build it, measure it, and be willing to throw it away. Five obvious fixes in this project measured worse than doing nothing.

Honest limits

• Made-up athletes throughout, and an AI reviewer judging an AI writer. Both sides share blind spots we cannot see from inside.
• The reviewer makes its own mistakes. In one verdict it got two separate facts wrong about the same reply, including claiming a session was missing that the fixture plainly contained.
• We have not built the truthfulness question yet. Until we do, our composite cannot see the failure we care most about.
• The noise floor was measured on one unchanged prompt, not across many, so treat three points as a working figure rather than a measured constant.
• Our review set was written by the same people who wrote the coach's instructions. It is held back from tuning, not from authorship, and a set written by someone else would be a stronger check.

Curious what a coach built this way actually says to you? Get early access.