A Chatbot Beat Qualified Trainers on Nine Questions. Does That Make It a Coach?

About this post: we had no involvement in this study, no connection to its authors, and no role in funding or reviewing it. It is one peer-reviewed study of nine questions and one model version, and its authors describe their findings as exploratory and hypothesis-generating. Anything below about our own system is our own measurement on made-up athletes, and is labelled as such.

Most claims about AI coaching are made by people selling AI coaching, including us. So it is worth paying attention when a university runs the comparison properly and publishes the whole thing.

Brecht D'hoe, Daniel Kirk, Jan Boone and Alessandro Colosio, working at Ghent University's Department of Movement and Sports Sciences with colleagues at King's College London and in Saint-Etienne, did exactly that. Their paper, "ChatGPT outperforms personal trainers in answering common exercise training questions", appeared in the Journal of Sports Science and Medicine on 1 February 2026. It is open access, and the summary below is ours; the work is entirely theirs.

What they actually did

The design is the interesting part, more than the headline.

They recruited personal trainers through Ghent University and the European Register of Exercise Professionals. Everyone had to hold at least an EQF level 4 qualification and be actively working as a trainer, which rules out the self-appointed. Each was asked for the questions clients ask them most often, together with their own answers, written the way they would write them for a client, in Dutch, French or English.

Forty-seven submissions came in. After exclusions, for answers that were not usable as a reply to a client, or fell outside 80 to 300 words, or did not concern exercise training, nine questions from nine different trainers survived into the study.

Those nine went to ChatGPT 3.5 in their original wording, on default settings, with no extra context and no follow-up. Each was asked once and the first answer recorded.

Then the grading. Every set of answers was scored by 27 people: 18 personal trainers who graded all nine questions, and nine subject experts per topic, each holding a doctorate, working at a research institution and publishing on that specific topic, who graded only the question in their own field. The order of the answers was randomised so graders could not learn a pattern, and nobody was told the study involved a chatbot until after they had finished. Everyone, trainers and graders alike, was then told and offered the chance to withdraw their contribution. Nobody did.

Three scores per answer, each from 0 to 10. Scientific correctness, meaning how well the answer matches what the field actually knows. Comprehensibility, meaning whether the person asking would understand it. Actionability, meaning whether they could do anything with it. The overall score is the average of the three. Group differences were tested with permutation tests over 100,000 simulations, with a second pass using the median instead of the mean to check that outliers were not doing the work.

Data collection ran from March to July 2024, which fixes the model version in time.

What they found

Question 9 leaned the same way with weaker evidence (p = 0.054). Questions 4 and 5 came out level. Broken down by criterion, ChatGPT was ahead on scientific correctness for five of the nine, on comprehensibility for six, and on actionability for five.

The result that gives the paper its title is the one in the other direction, or rather the absence of one: on no question, and on no individual criterion, did the trainers' answers score higher than the chatbot's. The authors note that a 2023 study of nutrition questions, which compared a chatbot against dietitians, found the same one-sided pattern.

One detail worth keeping. ChatGPT's answers averaged 241 words against the trainers' 140, so it wrote more. But when the authors ran a standard readability measure over both sets, they found no significant difference between them. The comprehensibility gap was not simply the chatbot using simpler English.

The finding we keep thinking about

Buried in the reliability analysis is a result that has nothing to do with who won.

Agreement between graders was good overall, but the single weakest spot in the whole study was scientific correctness on ChatGPT's answers. Among the grading trainers, that was the one combination where agreement dropped away. Among the subject experts who could be compared on one topic, agreement was generally higher for the human answers than for the chatbot's, across all three criteria. The picture is not entirely one way, and the authors report a separate consistency analysis that leans the other direction, but correctness on machine-written text is where the raters pulled apart.

That is a measurement problem, and we recognise it, because we have one of our own. Our coach's replies are scored by an AI reviewer, and we found that it cannot see a fabricated detail: an invented specific reads as specific, so it passes. We wrote that up in what if the test is wrong, not the AI. Fluent, confident, well-structured text is hard to mark wrong, whether the grader is a professor or a model. The Ghent paper puts a number on how hard, with 27 humans doing the grading, which is 27 more than we had.

What the study does not say

The authors are careful about this, and we will be too.

• Nine questions. They planned for seventeen, analysed nine because of the response rate, and call the work exploratory and hypothesis-generating rather than definitive. They note the design still produced 486 individual ratings.
• Mostly one topic. Five of the nine questions were about training for fat loss. The others were training frequency, motivation, training around pain, and what time of day to train. This is general fitness advice, not endurance coaching, and none of it involves a training plan.
• One model, one version, one attempt. ChatGPT 3.5, default settings, asked once. The authors expect newer models to behave differently and recommend that future studies record the model, its release date and the prompting protocol as a matter of method. In a field where the subject changes every few months, that is a better recommendation than it sounds.
• Answering is not coaching. The study measures informational support: how good an answer to a general question is. It does not test whether a model can hold a plan together across a season, read someone's actual training, or know when to say nothing. The authors frame the role as supporting trainers and trainees, not replacing them.

What we changed because of it

Three things, and we are being precise about which are done.

A correctness criterion, which we did not have. Our AI reviewer asks five questions about every reply our coach writes, and until we read this paper none of them asked whether the reply was true. Theirs leads with scientific correctness. We had independently found the gap by reading replies, but the Ghent rubric is a worked, peer-reviewed version of the question we needed to ask, and it is what our own is being modelled on. That work is in progress, not finished.

More than one rater, and agreement reported rather than assumed. They used 27 graders and published intraclass correlations. We use one model making one pass, and we report a noise floor measured by re-running an unchanged prompt. That is weaker, and we now treat it as the standard to reach rather than as good enough.

Randomised order. They shuffled the answers so graders could not learn a pattern. We already test one of our models for sensitivity to option order, described in the AI that never writes a word, and we had not applied the same suspicion to our own reviewer. We should have.

Where the pipeline goes next

This section is our plan, not a result. None of it is measured yet.

Jev: a panel, not a single opinion. Jev is a decision model that never writes a sentence. It returns a choice and a probability, which is why it is cheap and fast enough to run constantly. A rubric made of small classifiers, each answering one criterion with a calibrated probability, is structurally much closer to a 27-grader panel than one chatbot writing a verdict in prose. You can run it many times over for almost nothing, which means agreement becomes something you compute rather than something you claim.

Tools: the study's model had none. ChatGPT answered those nine questions from the question alone, with no context about the person asking and nothing to look anything up in. Our coach does not have to work that way. The engine hands it structured facts and a set of tools, which lets us split the work: a general question takes the knowledge path, where this paper shows a model is already strong, while a question about your Tuesday goes through the engine, which supplies the real figures. That split is also the honest answer to our own worst finding, where the writing step manufactures a plausible training week out of vague facts. A model that is handed your week does not have to imagine one.

Data: the part the study could not test. The nine questions had nobody attached to them. "How often should I train" has one answer in general and a different one for someone whose last six weeks we can actually read. That shift, from a correct general answer to a correct answer about you, is the entire product, and it is also where our measurements say the risk sits. Which is why the correctness criterion comes first and the personalisation is built on top of it, rather than the other way round.

Honest limits

• This is one study. It has not been replicated, we did not reproduce it, and we have no independent view of its data beyond the published paper.
• Nine questions, nine trainers, one model version, and five of the nine questions on a single topic. The authors' own framing is exploratory.
• The population is gym-based personal training. We work in endurance training, and we have no evidence the result transfers.
• Everything in the last two sections about our own system is either in progress or a plan, and the figures we quote about our own coach come from made-up athletes judged by an AI reviewer, not from anyone's real training.
• We are not a neutral party. A paper reporting that a chatbot answers training questions well is a convenient one for us to write about, and you should weigh it accordingly.

Want the general answer to be about your actual training? Get early access.