After Five Episodes, What Still Has to Be Human?

Every few episodes, we stop interviewing and start synthesizing. This is our sixth Reflections episode, covering five healthcare AI conversations: Sarah Rossetti, Jeff Smith of ONC, Hugo Campos, Fred Bennett, and Zak Kohane.

Usually we go looking for one thread to tie a block together. This time we decided not to force it, and that decision was its own small relief. The honest version is that these five didn’t resolve into a single slogan. But a question did keep returning, asked five different ways: as AI gets genuinely capable, what still has to be human, and what makes any of it trustworthy?

The human is still the sensor. Sarah Rossetti’s CONCERN system is the cleverest quiet idea we’ve seen in a while. Instead of modeling the patient, it models the nurse. The signal isn’t the vital signs, it’s the density and timing of the nurse’s documentation. A nurse charting at 3 a.m. on a patient who should be asleep is telling you she’s worried, before the vitals move. The system reads a trace nurses already leave (pure digital exhaust, no new documentation burden) and cut instantaneous mortality risk by 35.6% in a multi-site trial. Leon’s frame for this kind of work: real-world evidence is whatever the tide left on the beach. Most of it is seaweed and driftwood; the cleverness is finding the rare treasure, or turning the seaweed into fertilizer. The opposite of that craft is “method slop”: sophisticated processing that produces tidy p-values and no real meaning. The human enters twice, as the sensor and as the designer of the method. The AI is the enabler.

Empowerment and black boxes arrive together. Hugo Campos is the patient advocate who stopped waiting for the system to change and built his own tools on his own data, including an open-source server that pulls his medical record. He’s also refreshingly honest that he doesn’t fully understand what he built. That’s the tension in patient-directed AI in one person: the same natural-language tools that finally make a non-engineer feel capable (“the first power tool that didn’t make me feel stupid,” as a friend of Leon’s put it) also make it easy to assemble magical black boxes. Steve has watched the downside up close: someone close to him, fluent in data but not in clinical reading, used AI on her own records and landed 180 degrees from the truth. Empowerment without context is its own risk.

Trust is the actual product. Fred Bennett, building patient-side ambient AI, gave us the line we’ll keep using: not a Minimum Viable Product, a Minimum Trustable Product. The first time it fails a patient, they’re gone. It maps onto SEAT (Safety, Efficacy, Equity, Trust): drop the T and the rest never gets a turn. And the unglamorous business question underneath it, the one Leon teaches at Yale: is this a company or a component? In healthcare, no billing code means no money. The same tool that dies as a standalone app can thrive wired into care navigation, where the codes already exist.

You measure values, you don’t ask for them. Zak Kohane closed the loop. The man who once asked why EHR functions can’t be modular like iPhone apps (the question behind SMART on FHIR) is now asking what values our clinical AI models actually act on. You can’t ask a model, any more than you can ask a person and trust the answer; the explanation is a story told after the decision. So his group measures: give a model choices, flip its role from endocrinologist to insurance reviewer, and watch the recommendation reverse. Meanwhile the knowledge those models read is already being “bombed” with injected signal designed to be picked up as truth. The defense isn’t only clean training data, it’s monitoring behavior. Or, as Leon put it on the way out: who watches the watchers? Apparently we’ve volunteered.

After six Reflections blocks, that’s the most useful thing we can tell you. The technology keeps clearing bars. What stays scarce is the human judgment about what to measure, what to trust, and who it’s all actually working for.

Listen to the full conversation:

You Might Also Enjoy

• S1E39 with Sarah Rossetti, RN — The nursing-informatics conversation behind this block’s “model the clinician, not the patient” idea, and the CONCERN early-warning results in full.

• S1E41 with Hugo Campos — The patient advocate on building your own tools on your own data, two-AI verification, and the honest limits of “I don’t know what I built.”

• S1E43 with Zak Kohane — From SMART on FHIR to the Human Values Project: who controls healthcare data, and how to measure the values a model is acting on.