What Does 87% Mean? A Clinician’s Guide to AI Benchmarks

Written by Nader Absi, MD

Open almost any comparison of clinical AI models and you will see a familiar image: a bar chart with a clear winner. One model scores 87%. Another scores 82%.

The useful question comes before the ranking. What is the 87% a percentage of?

A benchmark score looks precise, yet it can leave much of what matters to a clinician unanswered. Before reading the bars, look at the cases, reference answers, grading method and test conditions. An exam vignette, a patient conversation and a longitudinal health record can all produce a percentage. They measure different capabilities.

What a benchmark measures

A benchmark is a standardized test used to compare systems under a defined protocol. It needs a task, a set of cases, a reference standard and a scoring method.

Consider the simplest example. A model receives 1,000 medical multiple-choice questions. The answer key defines correctness. The final score is the percentage answered correctly.

That number is valid for the test it describes. It tells us how the model performed on those questions under those conditions. It does not tell us whether the model can gather a history, notice missing information, use a tool correctly or communicate uncertainty to a patient.

This is where clinical benchmarks differ from many general AI tests. Clinical usefulness depends on the process as well as the final answer. The seriousness of an error matters. So do the patient population, the care setting, the language and the decision the model is meant to support.

For that reason, I read a benchmark as a study of one defined capability. The leaderboard is only the quickest summary of the method behind it.

The benchmark families you will encounter

Medical AI evaluation began largely with structured knowledge tests. MedQA, MedMCQA and medical subsets of MMLU use examination-style questions with one expected answer. They remain useful for measuring medical knowledge and some forms of reasoning. Their format also removes much of the work that happens before a clinical decision. The case is already written, the relevant details are usually present and the answer options are provided.

Literature benchmarks test a different skill. PubMedQA, for example, asks models to reason from biomedical abstracts. That is relevant to evidence interpretation. It provides limited information about symptom triage or clinical documentation.

Benchmark suites combine several task types. MultiMedQA, introduced with the Med-PaLM research, brought together professional examination questions, biomedical research questions and consumer health questions. It also used physician assessment of open-ended answers, including factuality and potential harm.

More recent benchmarks increasingly use realistic conversations and case-specific rubrics. HealthBench contains 5,000 multi-turn, multilingual health conversations developed with 262 physicians who have practiced in 60 countries. Each case has physician-written criteria describing what a strong response should include or avoid. This allows the score to reflect omissions, communication and safety concerns that cannot be captured by an answer key alone.

HealthBench Professional, released in April 2026, focuses on tasks physicians brought to ChatGPT for Clinicians across care consultation, documentation and medical research. MedHELM takes a broader approach, organizing medical AI work into a clinician-validated taxonomy of 121 tasks linked to 37 evaluations.

The direction is moving beyond response generation. AgentClinic places a model in an interactive clinical simulation. The agent must gather information through dialogue, request measurements and work with multimodal inputs before making a diagnosis. It includes specialist cases and multilingual scenarios. PhysicianBench evaluates long EHR workflows that require retrieval across encounters, clinical reasoning, actions inside the record and documentation. LongMedBench, released as a preprint in July 2026, tests whether agents can reason across repeated admissions and an evolving patient history.

These newer designs test more of the clinical workflow, while leaving the reader with the same work of examining case selection, grading and applicability.

How a clinical benchmark is built

A credible benchmark begins with a clear intended use. The developers need to define the user, the task and the consequences of failure. A consumer health benchmark should look different from one evaluating medication ordering inside an EHR.

Cases are selected or created to represent that task. They may come from examinations, literature, de-identified records, clinician-authored scenarios or real interactions. Their source affects what the score can support. Cases from one country or specialty may provide weak evidence elsewhere.

The reference standard is often the hardest part. A factual question can have a verified answer. Open-ended clinical responses require judgment. Several clinicians may review the same response, work from a rubric and resolve disagreement through adjudication. Good reporting describes the reviewers, their specialties, whether they were blinded to model identity and how often they agreed.

Automated grading adds another layer. Language models are increasingly used to decide whether a response satisfies rubric criteria. This makes large evaluations practical, but the grader becomes part of the measurement system. It should be tested against independent physician judgment, and possible preference for related model families should be examined.

A recent German preprint called MedQADE makes this problem unusually clear. It contains 3,800 open-response items annotated by practicing physicians and automated evaluators. The strongest evaluator reached agreement close to the physician ceiling, but the scores concealed an important behavioral difference. Physicians abstained more as questions became difficult. The automated evaluators still issued a definitive judgment in every case. The study also found model-family preferences in grading.

Agreement statistics alone may therefore miss clinical caution. A grader can reproduce many physician labels while responding differently to uncertainty.

The scoring method comes next. Accuracy may be suitable for fixed-answer questions. Other tasks may require sensitivity, specificity, rubric scores, citation checks, execution-based checkpoints or ratings weighted by potential harm. The metric should match the intended capability and should not reward irrelevant features such as excessive response length.

The test protocol also matters. Prompts, tools, model settings, retries and model-specific adaptations should be disclosed. Generative systems can produce different answers to the same input, so repeated runs and variability may be more informative than a single score.

For diagnostic and prognostic prediction models, TRIPOD+AI recommends reporting confidence intervals, subgroup performance and error analyses. Generative AI evaluation uses different methods, but the same demand for transparent reporting remains useful.

How to read the graph

When I review one of these charts, I rarely begin with the tallest bar. The benchmark name and task come first, because the same percentage can mean factual accuracy, rubric completion, diagnostic sensitivity or success in an EHR workflow.

After that, I look at the cases and reference standard together: their source, sample size, specialty mix, language and date, along with who defined correctness, how disagreements were handled and whether the grading process was independent. This usually tells me whether the ranking answers a question I care about.

The metric comes next. Higher is not always better. Error rates and harmful-response rates move in the opposite direction, while a composite score may hide which criteria received the most weight.

Uncertainty changes how the bars should be read. A two-point difference may represent many additional correct answers in a large dataset, or one case in a small one. Confidence intervals and repeated runs help show how stable the difference is.

The protocol deserves the same attention. A model with access to retrieval and a closed-book model are completing different tasks. Best-of-five scoring should not be placed beside a single attempt without a clear label, and the exact model versions and evaluation dates should be visible.

In my own reviews, I often learn more from the error table than from the ranking. An average can conceal underperformance in emergencies, rare presentations, particular specialties or underrepresented populations. The location and severity of errors may carry more clinical meaning than the overall count.

Human comparisons require the same care. “Doctor performance” could refer to medical students, residents, general practitioners or specialists, each working with different time limits and access to tools.

Where benchmark scores mislead

Public benchmarks can remain online for years. Questions, answers and decisions may enter future training data, allowing familiarity to inflate performance. Rolling or time-separated datasets can reduce this risk, although complete exclusion from training data is difficult to prove.

A benchmark can also reward the wrong behavior. A clinically useful system may need to ask for more information, state that evidence is insufficient or recommend escalation. A scoring system focused on confident completion can penalize appropriate restraint.

Evidence-retrieval systems need additional checks. A correct answer attached to a broken, outdated or irrelevant citation should not receive full credit. Evaluation should verify that sources exist and support the claims made.

Geography and language deserve attention too. Strong performance on American English examination questions offers limited evidence about use in Beirut, Riyadh or Munich. Different guidelines, documentation habits and patient populations can change the task.

The benchmark developer’s relationship to the systems being compared should be visible. A company can evaluate its own model carefully. Disclosure helps readers interpret the design, case selection and published ranking.

A benchmark result describes performance on a defined test at a defined time. Clinical readiness requires evidence in the intended population and workflow, followed by prospective evaluation where the system will be used.

Twelve questions before trusting the score

• What capability was tested?
• Who built and funded the benchmark?
• Where did the cases come from?
• Do they resemble the intended patients, users and setting?
• Who created the reference answers or rubrics?
• Were graders blinded, and how was disagreement handled?
• If a model graded the answers, how was that grader validated?
• What does the metric reward?
• Were prompts, tools, retries and model adaptations disclosed?
• Are sample size, uncertainty and repeated runs reported?
• Are high-risk errors and subgroup results visible?
• Could the test material have entered training data?

Benchmarks give us a common way to compare systems and study where they fail. Their value depends on reading them with the same discipline used for clinical evidence: population, methods, outcomes, uncertainty and applicability.

At DR. INFO, we believe a benchmark chart should begin an evidence review. The information needed for that review should be easy to find.

When you see a clinical AI benchmark chart, where do you begin?