The demo is always the same. A clinician finishes a visit, the software writes the note, and it lands in the chart. No typing. The pitch is autonomy: the AI does the work, the clinician moves on.
The version clinics actually adopt is narrower than that, and the gap between the two is where most clinical AI projects stall. It is worth being precise about why.
The accountability doesn't move
A clinician signs the note. If the note is wrong — a missed finding, a copied-forward problem that resolved months ago, a medication that isn't current — it is still their signature on it. Software that drafts the note does not change who is answerable for its contents.
So the useful design question is not “can the model write the note?” It is “can we make review fast enough that a clinician is willing to do it every time?” Those are different products. The first removes a step. The second keeps the step and makes it cheap.
Draft, don't file
There is a hard line between a tool that proposes and a tool that acts. A tool that proposes puts a draft in front of a person and waits. The worst case is a bad draft that gets rejected — annoying, not dangerous. A tool that acts writes to the record on its own. The worst case is a bad entry that nobody caught.
Most models, in most clinical settings, have not earned the second mode. That is not a permanent verdict, but it is the current one, and a responsible rollout treats “filed only on approval” as a requirement rather than a phase-one compromise.
Evaluation is the product
A vendor benchmark tells you how a model did on someone else's cases. It does not tell you how it will do on your patients, your specialties, your documentation habits, or the edge cases your panel actually contains.
The thing that makes clinical AI safe to run is an evaluation set built from your own de-identified cases, with expected outputs, that every model change and every prompt change has to pass before it ships. If a change makes the summaries worse on your set, it does not go out. Teams that skip this step are not measuring the thing they are accountable for.
What good looks like
- Narrow scope — a few document types, a few specialties, not “all clinical writing.”
- Drafts are visibly drafts, and nothing reaches the record without a person approving it.
- The draft and the final version are both retained, so the difference is auditable.
- There is an evaluation set of real cases, and changes are gated on it.
- There is an off switch, and someone whose job is to use it if the outputs drift.
The unglamorous version wins
AI that assists is a slower pitch. It saves less time than the demo implied, and it keeps a human in the loop where the demo removed one. It is also the version that survives contact with a real practice, because it is designed around the constraint that actually governs clinical software: a licensed person is responsible for what goes in the chart, and no amount of model accuracy transfers that responsibility somewhere else.