HealthBench Professional GuideThe short answer
HealthBench Professional is deliberately designed to remain challenging. Its selected tasks therefore answer a different question from a random sample of ordinary clinician use. The distinction is central to interpreting its score. This guide examines selection, use-case composition and repeated sampling, then develops a claim that preserves the benchmark’s stress-testing value without treating it as a routine clinical failure rate.
Follow selection from candidate pool to benchmark
The paper describes a final set of 525 tasks selected from 15,079 candidates. Review and stratified sampling emphasized difficult examples, including deliberate adversarial testing. The resulting distribution is a design choice intended to expose remaining limitations of strong systems, rather than a claim about how often each scenario occurs in practice.
The appropriate first question is therefore what behavior the selected cases stress. A moderate score can reflect performance on an intentionally demanding set without estimating the share of ordinary interactions that would fail. Conversely, a strong score on the selected set does not establish performance on every workflow excluded from it.
Keep source slices visible
The paper separates good-faith typical, good-faith difficult and red-teaming difficult examples. These slices encode how the task arose and how it was rated during construction. They are not interchangeable categories of disease severity or direct measures of patient risk. Their meaning comes from the benchmark’s collection protocol.
An evaluation report should retain those slice identities next to the aggregate. If a system improves primarily on adversarial examples, that is a different pattern from uniform improvement across ordinary and difficult interactions. The overall mean is useful, but its interpretation becomes more specific when the source of the gain is visible.
Do not isolate use-case difficulty from its composition
Appendix Table 2 reports 91 red-teaming difficult tasks among 142 writing/documentation tasks, compared with seven among 147 research tasks. Those use cases therefore have different selected source mixtures. A direct comparison of their scores combines differences in task content with differences in the cases chosen for evaluation.
Our inference is that use-case bars alone cannot identify which activity is inherently harder. A more informative analysis examines the cross-classification of use case and source slice, with the relevant denominators. Sparse cells should remain visible; they are evidence about the limits of the comparison, not an invitation to invent stable estimates.
Distinguish repeated responses from new tasks
The main study comparisons average repeated samples for each task. Repeating generation helps characterize variability for the same examples, but it does not create additional distinct clinical scenarios. A report should separate the number of tasks from the number of generated responses so that its evidence volume is not overstated.
The paper’s statistical treatment recognizes examples as the paired comparison unit. When designing a new evaluation, preserve the structure of the actual data rather than treating every response as an unrelated patient encounter. The relevant uncertainty depends on what was sampled and what was repeated.
Write a conclusion about selected-condition performance
A defensible conclusion identifies the selected task distribution and the evaluated system configuration. It can describe performance under those intentionally difficult conditions. Estimating routine use would require a defined target population and evidence about how its task distribution relates to the benchmark; a score alone cannot supply that relationship.
The official release and paper make the intended professional scope clear, while the data card documents the fields needed to inspect it. Use those artifacts to build an explicit coverage map for your application. The benchmark contributes evidence where the tasks align, and the unmatched workflows define the next evaluation questions.
References & further reading
These original sources support the methods discussed here. Our suggested planning steps are editorial guidance, not an endorsement by the source authors.
- HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats ↗Rebecca Soskin Hicks and colleagues · OpenAI. Primary sampling, length adjustment, model/harness comparison and composition tables.
- HealthBench Professional official dataset card ↗OpenAI. MIT metadata, field definitions, contamination request and internal-evaluator limitation.
- Making ChatGPT better for clinicians ↗OpenAI. Dated announcement introducing the benchmark; separate from the subsequent arXiv version.
Published by Arcophos. Educational material, not clinical advice or a claim of regulatory compliance. Read our editorial method.