General-Purpose LLMs like ChatGPT in Emergency Triage: What MTS-Bench Found

General-purpose language models such as ChatGPT are increasingly used by physicians and triage nurses, often as an informal way to sanity-check how urgent a case is. Uptake is now substantial: about two in three US physicians report using AI tools in clinical practice [1], and one in five UK general practitioners reports using an AI chatbot [2]. That use is growing faster than the evidence behind it. Whether these tools are safe for triage, a task where getting the urgency wrong can delay care, had not been tested against a structured clinical framework [3]. We built MTS-Bench to answer a simple question: does a general-purpose model, used alone, assign the right urgency, and does a retrieval-grounded clinical system do better?

The key finding: MTS-Bench found materially different triage-safety profiles for the systems it tested. It evaluated one standalone general-purpose LLM, ChatGPT (GPT-5.1), alongside DR. INFO, an agentic clinical assistant that retrieves evidence from a curated knowledge base. DR. INFO undertriaged 11.5% of emergency-case prompts; ChatGPT undertriaged 44.2% [3].

The difference was most important in time-critical presentations. DR. INFO made no Red-priority undertriage errors in the benchmark, including when a misleading referral statement was added. Its Manchester Triage System (MTS) retrieval configuration also responded more appropriately when vital signs, examination findings and laboratory results became available.

General-purpose LLMs such as ChatGPT can be useful for drafting, education and broad clinical support. This safety benchmark, however, found substantial undertriage when ChatGPT was used alone for structured MTS-based emergency triage [3]. The study is available as a preprint on medRxiv and has not yet been peer reviewed.

Why emergency triage is a hard safety test for AI

Triage decides how quickly a patient should be assessed. In the Manchester Triage System, that ranges from Red (immediate assessment) to Blue (non-urgent, up to 240 minutes) [4].

The two kinds of error are not equal. Undertriage gives a patient a lower priority than their presentation warrants, potentially delaying time-critical care. Overtriage uses more resource, but it does not create the same delay for a sick patient. A safe triage aid therefore has to be especially reliable at the high-acuity end of the scale.

This is why the MTS is a useful test bed. It is a structured framework with explicit flowcharts, discriminators and time targets. It asks a system to do more than produce plausible medical prose: it must apply a defined decision procedure to the information in front of it [3][4].

What MTS-Bench tested

Each case was presented in several clinically relevant ways:

• with or without vital signs, examination findings and laboratory results;
• with male and female patient versions; and
• with or without a misleading GP referral statement, designed to test whether the system would be anchored by an authoritative but incorrect initial framing.

The study compared three configurations:

• ChatGPT (GPT-5.1), the standalone general-purpose LLM evaluated in this study;
• DR. INFO Baseline, an agentic clinical assistant retrieving from a curated clinical knowledge base; and
• DR. INFO with MTS retrieval, the same system with the Manchester Triage System reference text available at the time of the query.

We tested GPT-5.1 through the API as the general-purpose comparator for two reasons. It is the closest available equivalent to ChatGPT, the general-purpose model clinicians most often reach for as an ad hoc triage aid, so it stands in for that class of tool rather than for any one product. And a single, well-specified model and prompt let us run the comparison quickly and reproducibly. This is a deliberate scoping choice, not a claim about general-purpose LLMs as a category: other models, versions and prompts can behave differently, and a recent nine-model study found wide variation in triage error between them [6].

Each system received 312 prompts. The primary analysis reports the 156 prompts without the misleading referral, so that baseline triage performance can be compared directly.

Comparative triage-safety results: DR. INFO and ChatGPT

The first table compares the primary safety outcome, undertriage, in the 156 prompts presented without a misleading referral statement. It separates the overall result from the Red-priority subset, where the MTS requires immediate assessment.

Without the misleading referral statement, both DR. INFO configurations undertriaged 18 of 156 prompts (11.5%). ChatGPT, the standalone general-purpose LLM tested here, undertriaged 69 of 156 (44.2%).

The difference in overall undertriage between ChatGPT and each DR. INFO configuration was statistically significant (Fisher's exact p = 1.0 x 10^-10). DR. INFO's remaining errors were generally in the more cautious direction, overtriage, whereas ChatGPT's were more often undertriage [3].

Safety by MTS priority: DR. INFO and ChatGPT

The overall rate can obscure where errors occur. This table breaks down undertriage by the benchmark's physician-verified MTS priority, from Red (immediate) to Blue (non-urgent).

There were only eight Red-case prompt permutations, so the estimate has a wide confidence interval. Even so, the observed errors in the standalone model were concentrated in the urgency categories where waiting is least tolerable [3].

Response to misleading referral framing

Clinical decisions rarely begin with a blank page. A referral note, preliminary diagnosis or prior clinician opinion can shape the first impression of a case. The benchmark tested this with a plausible but incorrect GP referral statement placed before the vignette [3][5].

Overall undertriage did not differ significantly for any system. At the Red priority, ChatGPT undertriaged 8 of 8 cases with the misleading referral, compared with 6 of 8 without it. Both DR. INFO configurations remained at zero Red undertriage [3].

This is a useful practical warning. A triage assistant should reassess the evidence, rather than simply reinforce the first label attached to a patient.

Response to objective clinical data: DR. INFO with MTS retrieval and ChatGPT

Triage is not a one-time verdict. A patient may be assigned an initial priority from their history and then reassessed when observations, examination findings or laboratory results become available.

The following comparison uses the same cases first without, then with, vital signs, examination findings and laboratory results. It shows how each system's undertriage rate differed between the two inputs; it does not establish why those differences occurred.

The MTS-retrieval configuration was the only one whose within-system difference reached the study's pre-specified significance threshold. Because it retrieves the relevant MTS flowchart and its thresholds at inference time, this configuration provides a framework-grounded way to assess the case; the benchmark itself does not isolate retrieval as the sole cause of the observed difference [3].

What an independent physician review found about DR. INFO and ChatGPT

The study did not rely only on automated scoring. A physician independent of DR. INFO reviewed the high-risk subset, blinded to both the system identity and the benchmark's gold-standard label [3].

ChatGPT responses were flagged as high risk in 107 of 312 responses (34.3%), compared with 24/312 (7.7%) for DR. INFO Baseline and 20/312 (6.4%) for DR. INFO with MTS retrieval. Within that high-risk subset, the reviewer confirmed a clinically dangerous under-call in 15 of 107 ChatGPT cases (14.0%) and in 0 cases from either DR. INFO configuration.

The independent reviewer described the residual DR. INFO disagreements as generally cautious over-calls, while the ChatGPT disagreements more often risked false reassurance [3].

What this safety benchmark means for clinicians using general-purpose LLMs

The finding is specific: in a benchmark of structured MTS-based emergency triage, the tested ChatGPT (GPT-5.1) configuration undertriaged the most time-critical presentations far more often than the retrieval-grounded systems, and its undertriage rate rose rather than fell when objective clinical data were added [3]. This describes the behaviour of one general-purpose setup on this benchmark, not the safety of general-purpose LLMs as a class.

That is not a verdict on every use of general-purpose AI in healthcare. General models can be useful for drafting, education and other clinical support tasks. But a task governed by an explicit, time-sensitive decision framework favours a system that can retrieve and show the rules it is applying. In this study that system also used a different base model from the standalone comparator, so the benchmark cannot separate the contribution of retrieval from the base model; the clean comparison is within DR. INFO, where adding the framework text improved safety on the same base model.

For nurses, physicians and other clinicians using a general-purpose LLM alongside triage work, the practical questions are:

• Can the system retrieve the clinical framework or guideline relevant to this decision?
• Does it cite the source used for its recommendation?
• Does it update its assessment when observations and test results change?
• Does it preserve clinician accountability rather than presenting itself as the decision-maker?

DR. INFO was designed around these requirements: it searches a curated clinical knowledge base, structures its assessment around the evidence retrieved and provides citations for clinician review. In MTS-Bench, adding the MTS reference text to that workflow was associated with the strongest overall safety profile of the three configurations evaluated.

What MTS-Bench does not show

This was a benchmark study, not a prospective trial in emergency departments or primary care. It tested text vignettes, not the direct observation, examination and clinical judgement involved in real triage.

It also has important methodological limits. Model outputs were sampled once per prompt; the 39 underlying cases produced several related prompt permutations rather than fully independent observations; and only eight prompt permutations contributed to the Red-priority estimate. DR. INFO was built on a different frontier base model from the standalone GPT-5.1 comparator, so the cross-system comparison cannot isolate retrieval alone. The benchmark was also created by the DR. INFO team, despite the safeguards of within-system comparisons and independent physician review. Independent replication and prospective clinical evaluation are the necessary next steps [3].

The bottom line

The clearest result in this benchmark is about adaptive triage. When vital signs and laboratory results were added to a case, DR. INFO with MTS retrieval improved significantly (19.2% to 3.8% undertriage, p = 0.005), while the tested GPT-5.1 configuration did not improve and moved in the opposite direction. On the same cases, that GPT-5.1 setup undertriaged more often overall, most at the Red and Orange priorities. These results describe the tested configurations on this benchmark, not general-purpose LLMs as a class [3].

The safer pattern in this benchmark came from DR. INFO, especially when it could access the Manchester Triage System at the point of use. For high-stakes clinical decisions, the goal should not be an AI that sounds confident. It should be an AI that can apply the relevant framework, expose the evidence behind its answer, and remain under clinical supervision.