While a clinical AI model can place near the top on medical licensing exams, it is still completely capable of – and often prone to – misjudging the patient it’s attempting to serve. For tools created to serve Medicaid beneficiaries, two things set that gap before deployment: the patients a model learned from, and the architecture that governs how it acts when a case moves past what it knows. A language model trained on the right population can still misjudge a patient. Keeping that patient safe is the work of the system built around the model.

Medicaid covers a population that concentrates comorbidity and tightly couples physical and behavioral health. KFF's analysis of claims data finds behavioral health conditions diagnosed in roughly one in five to one in four adult enrollees, physical chronic conditions rising to 61 percent of those aged 50 to 64, and more than 2.3 million adults carrying a serious mental illness diagnosis, who also show higher rates of co-occurring chronic conditions and substance use disorder. Claims data understate true prevalence. Add housing instability, transportation barriers, and limited English proficiency – each of which changes how a symptom should be read – and the patient looks little like the average case in a general training corpus.

As a result, patients receiving Medicaid benefits are not well served by these tools. A 2026 Nature Medicine evaluation of ChatGPT Health, OpenAI's consumer health tool, ran it through 60 clinician-authored vignettes across 21 clinical domains. It undertriaged 52% of gold-standard emergencies, telling patients with conditions such as diabetic ketoacidosis to seek care within 24 to 48 hours rather than go to the emergency department, while handling textbook emergencies like stroke and anaphylaxis correctly. It performed well in the middle of the acuity range and broke down at the edges.

Essentially, topping a multiple-choice exam had not prepared them to manage an actual patient – especially not Medicaid beneficiaries. So how can health plan and partnership leaders evaluating clinical AI tools against a backdrop of service to Medicaid beneficiaries select the right AI for this unique population?

Why scaling a language model cannot close the gap

An AI model learns by example. It performs well on the kinds of patients it saw often in training and poorly on the kinds it rarely saw. So when the people a tool serves look different from the people it learned on, in their mix of conditions, their social circumstances, and how their symptoms present, its accuracy drops. This is one of the most consistent findings in machine learning, and a Medicaid population is exactly where that gap opens.

Making the model bigger or smarter does not fix this. A larger model, more elaborate reasoning, or a search tool bolted on top can help it use what it already learned. None of them can give it experience with patients it never saw. The search-tool approach makes this clearest: the tool looks things up in a reference library as it answers, so if that library holds few patients like yours, it has nothing useful to pull.

In a 2025 Nature Medicine study, a model trained on real-world clinical records outperformed general commercial models across specialties. Its biggest lead came on the unusual, hard-to-place cases, the ones that do not match a common pattern. The ChatGPT Health results above show the other half of the problem.

Consumer tools are not what a health plan is choosing between. They are useful here only because they make the pattern easy to see. They also show why training data alone cannot settle the safety question. A model that has seen the right patients still behaves unpredictably at the edges of urgency, because that behavior comes from how a probabilistic system handles uncertainty, a property more data does not change. A clinically safe tool has to move the safety-critical decisions onto a different architecture, one that does not depend on the model judging the moment correctly.

Evaluating efficacy while ensuring patient equity

For a medical director, this reframes what to ask. A model's size and benchmark scores describe capability in the abstract, but those say little about whether the system has encountered patients like any one plan's members. Training population is one of three questions worth putting to any clinical AI, alongside what the system is optimized to do and how its safety-critical path is built. Each exposes a different way a tool can fail a complex patient, and training population sets the ceiling for the other two, since a model that never learned the population cannot reason or escalate its way past that limit. This is the category argument in practice: clinical AI for Medicaid calls for a different evaluative standard than general-purpose AI, and the gap between those standards is paid by the patients the model will meet.

Waymark built Compass against this standard. Compass is trained on more than 200,000 real Medicaid encounters, so the patterns it has learned come from the population it serves. Its safety-critical work runs on a separate, deterministic layer rather than the conversational model, and it acts on what it finds: arranging transport, conducting three-way pharmacy calls, booking appointments, coordinating benefits, and escalating urgent cases to a Waymark physician for immediate follow-up.

For a clinician, Compass reflects the patients they treat and escalates an urgent case to a Waymark physician for follow-up. For a health plan, it extends care management at population scale without adding staff, with the safety-critical steps handled by a layer built for that purpose. Two design choices sit behind both: train the model on the population the system serves, and keep the safety-critical path on an architecture built for reliability.

As health plans bring AI into care management, choosing which system to trust is a clinical decision before it is a purchasing one. The complexity that defines a Medicaid population is also what breaks a model built for a different one, and no amount of engineering layered on top can recover or compensate for what the model never learned. That makes the training-population question the one that protects patients, and the leaders making these calls now are setting the standard to which the next wave of clinical AI in Medicaid will be held.