Can an AI That Never Writes a Word Read Your Coaching Messages?

About this post: every message in our tests was written by us, and the labels were set by AI reviewers. The figures describe those made-up messages. They are not measurements on real athletes.

Most AI coaches are chatbots: you type, a large language model writes back. Jev, from a company called TypeSafe, is a different kind of tool. It never writes a sentence.

What it does

You send Jev a small description of the situation and a list of questions. It answers with a choice and a probability for each option ("70 percent this, 20 percent that"), or a probability of yes for a yes-or-no question. That is all. No paragraphs, no streaming text.

In our setup, Jev reads each athlete message and answers 11 questions: what is this message about, what change is being asked for, which day, how many minutes, and a handful of yes-or-no questions such as "does the athlete say they feel great?" or "do they want to train harder?". Ordinary code then decides what to do with those answers, using fixed thresholds. If Jev is unsure, the message goes to the slower, fuller AI path instead. Jev never sets training load, dates or hours. Those stay in code.

The idea is simple. If a cheap, fast reader can sort most messages correctly, an expensive chatbot only has to handle the hard ones.

How a message becomes a decision

Take a message like "legs are shot, can I move Thursday's ride to Saturday?". Four steps follow.

1. Jev gets a small summary, not the whole file. It sees the message and a short, named description of the situation. Numbers and dates are not its job.
2. Jev answers the 11 questions. Four are about the content: the intent, the change being asked for, the day and the minutes. Seven are yes-or-no checks: illness, pain, a medical red flag, very tired, feels great, wants to train harder, and agrees with what the coach said last. For our example message, the useful reads are "very tired: likely yes", "change: move a session" and "day: Saturday".
3. Code turns the answers into a route. A fixed set of thresholds sends the message to a safety response, a plan change, a suggestion, an explanation of today's plan or the slow path. Whether Saturday is allowed is a question for the training engine, not for Jev.
4. A doubtful read never becomes a guess. If Jev's answer fails, or its own confidence is low, the message goes to the slow path, the full language-model route.

The split is deliberate. Jev does the one thing it is good at, which is sorting what someone meant into a few named boxes. Everything that touches a number, a date or a limit stays in code that can be read and tested.

The test

We wrote the plan before running anything and then froze it. The test set was 194 made-up coaching messages, 97 in English and 97 in Dutch. Two AI reviewers from different model families labelled every message without seeing each other's answers, and a third settled disagreements. Each reader saw each message three times.

We compared Jev with Anthropic's Claude Haiku 5.5, a small general language model, reading the same questions.

The accuracy gap is unlikely to be chance (p = 0.0014). Haiku also had 24 failed runs out of 582, mostly network errors, and we counted those as wrong. Its time includes retries.

Does it favour the first option?

Jev's makers say it can lean toward the first choice in a list. We checked by shuffling the order of options. We expected Jev to flip more. It flipped less, 11 times against Haiku's 40. We had guessed the opposite, and we report it as we found it.

What Jev is known to get wrong

TypeSafe's own description lists where a model like this is weak: very literal reading, numbers and dates, indirect wording, a lot of irrelevant detail, a lean toward the first option, and text written to fool it. That list is why numbers and dates stay in code, and why the order check above was part of our plan.

The weakness that matters most for coaching is the joke. Messages can hide a warning sign inside humour or understatement, with no symptom word in sight. In our set, 74 of the 194 messages contained a medical warning sign. Jev sent 72 of them to the safety response and missed 2. Both were sarcastic English messages: one joking about struggling to breathe on a flight of stairs, one making a casual remark about the colour of urine. Jev put the chance of a warning sign at roughly one in four, just under the line that triggers action. A list of keywords cannot catch a joke that contains none.

Two misses in 74 is a rate of 2.7%, and with a set this small the honest range is from under 1% to about 9%. That is why we do not call Jev safe, and why a second layer always sits above it. Our post on what an AI coach can and cannot do shows what we built on top of Jev's own doubt.

Honest limits

• The questions are Jev's own wording, copied into Haiku's prompt. That favours Jev.
• Jev's price comes from third-party listings, not from a contract.
• We measured speed from a laptop in the Netherlands.
• The messages are made-up, and real athletes are messier.
• Jev is not safe enough to be the only reader for health warnings, as the two misses above show.
• We did not test a larger model, so we cannot say whether one would beat Jev.

What we take from it

For the plain job of sorting everyday coaching messages, a cheap model that only returns choices and probabilities did better than a small chatbot on accuracy, speed and price, on our made-up messages. That is a promising result for one narrow task, not a verdict on AI coaching.

Interested in how pace24 reads athlete messages? Get early access.