What Happens When You Test an AI Coach Instead of Trusting It?

Start with the honest question

Most pages about AI coaching tell you what the AI can do. We would rather show you how we test it: we try things, measure them, publish what fell short and keep building on what worked. Right now we are improving the coaching engine around Jev combined with a large language model, and the early results are promising.

We build a coaching engine, and we run our own studies on made-up athletes and made-up messages. No real athlete data ever goes to an AI model in our lab. Here is what we learned about three jobs a coach does.

Job one: reading what the athlete says

Athletes write things like "legs are shot, can I move Thursday's ride to Saturday?" Before anything else, a system has to work out what that means: a tiredness report, a schedule change, a request for something harder?

We tested two readers on 194 made-up messages, half English and half Dutch, each read three times. One was Jev, a model from a company called TypeSafe that never writes sentences. It answers fixed questions with a choice and a probability. The other was Claude Haiku 5.5, a small general chatbot.

• Jev sorted 174 of 194 correctly (89.7%). The chatbot got 156 (80.4%).
• Jev's slowest replies took about 0.3 seconds. The chatbot's slowest took several seconds, and some failed.
• Jev cost about $0.16 per thousand messages. The chatbot cost about $1.10.

There are caveats. The questions were written in Jev's style, which favours it. Its price comes from third-party listings. The messages are made-up, and real ones are messier. But for the plain job of sorting everyday messages, a small, cheap, narrow model did better than a chatty one.

Jev is easy to mistake for a smaller chatbot. It is not. It never writes a reply: it answers a fixed list of questions, and ordinary code decides what to do with the answers. We explain the questions, the steps and where Jev failed our test in a post about Jev on its own.

Job two: deciding how much to train today

This is the job we would least hand to a chatbot, and we tested it.

We gave a language model a made-up athlete's last 28 days and the coming plan, and asked it for numbers only: the most load allowed today, whether a hard session is OK, and whether to go easy. We compared it with a rule-based engine across 21 scenarios, played forward day by day with an obedient simulated athlete.

The engine passed 10 of 21 checks. The chatbot passed 7. Neither was a clear winner.

The pattern mattered more. Over 267 decision days, the chatbot said to keep the day easy on 236, or 88%. Its typical cap was simply the athlete's usual day. It passed every check that says "do not do more than before." It failed every check that says "let a healthy athlete follow the plan."

The effect on fitness was clear. An athlete training at 80% of the allowed limit saw their fitness score climb from 49.7 to 82.4 under the engine and slip from 49.7 to 43.9 under the chatbot. A coach that is always careful and never helps you improve is not yet coaching. That is the gap the engine's rules are there to close.

This is one model, one prompt, one run, so it is a lead and not a verdict. A different prompt or a plan-writing task might behave differently. But it matches what you would expect from a system that is rewarded for sounding careful.

Job three: talking to the athlete

This is where a language model earns its place. Explaining why today is easy. Rewording a plan in plain language. Answering "why am I doing intervals this week?" Keeping a consistent voice.

We did not run a head-to-head test on conversation quality, so we make no claim about it. What we did decide is that the model talks and chooses inside a limit set by code. It may go lower or pick a different workout format. It cannot raise the limit.

Why we keep the numbers in code

Three reasons came out of our work.

You can read the rules. Every limit is a visible rule. We audited them one by one against the research, and several turned out to be weak, mis-sized or even dead (one rule meant to block hard days on two days running could never fire). Finding that in rules is easy. Finding it inside a model is not.

You can test them. Some things must always be true: adding training to your last week should never make today's limit higher, and saying you feel great should never lift a cap. We can check those over thousands of random histories.

You can replay them. The same history gives the same answer, so we can try a change on thousands of simulated athletes in seconds.

Where the AI struggles

Humour and understatement. In a second study, we wrote 73 messages in which warning signs were hidden in jokes and metaphors. The cheap reader missed 16 of 35, and the chatbot missed 12. That is exactly why no single AI reader is ever the only layer between an athlete and a safety response. It also pointed us to our best result so far.

What we are building: a cheap reader with a second opinion

Jev does not only give an answer. It gives a probability. On the disguised messages it missed, it was rarely confident either way: the probability of a warning sign usually sat in a grey zone, between 0.25 and 0.5, just under the line that triggers action.

So we tried something simple. Jev reads every message. When its own probability lands in that grey zone, or its confidence is low, a larger language model reads the message too, and the more cautious of the two readings wins. We wrote the test before running it, on a fresh set of 60 disguised messages the design had never seen.

The result held up:

• Of the 29 messages Jev alone missed, the second opinion caught 11 (31 of 60 became 42 of 60) and lost none.
• It did better than the larger model on its own (34 of 60), because the two readers miss different messages.
• It did not ask for a second opinion any more often on harmless messages than the simpler confidence rule did (11.6% in both cases).
• On everyday messages in a separate test, it asked for a second opinion about 5% of the time.

It also fell short in one place, and we say so. Our cost target was missed on that test, because most messages in the set were disguised warnings by design, so the larger model was called often. How much it costs on real traffic depends on how many messages land in the grey zone, which is what we measure next.

This is still early. The messages are made-up, the labels come from AI reviewers and a person's audit is pending. But it is the first time a cheap reader plus a larger model beat either one alone on a test we set in advance, and it is the direction we are building in.

The engine is improving too

The same habit applies to the training engine. After each round of tests we fix what the tests show. In the last round, one simulated athlete who ended a 14-week build phase a point below their start (68.5 against 69.5) now ends it at 75.4 after we filled a gap in the workout library. The fix is built, checked and sitting on our engine branch.

Who is it for?

From what we have seen, an AI coach is useful for people who want a plan that adjusts, answers in plain language and does not cost the earth. It is not a replacement for judgement about pain, illness or life events, and we did not test it against human coaches, so we cannot say how it compares.

What we have not measured

Real athletes. Long-term results. Human coaches as a comparison. Anything about conversation quality. Each of those needs a proper study, and we say so in our protocols.

Want to see how this fits together in the app? Get early access.