P065-Halamka-LinkedIn-Hero

The Registry Will, the Record Won’t: John Halamka on What Big Data Can Answer

John Halamka started with his hand. He runs an animal sanctuary with 410 animals, and a turkey bit him. His primary care doctor recorded an abrasion, and that’s what the code says. “The concept turkey bite was not communicated in any structured way in the data record,” he told us, “but yet it’s in the free text.”

John is president of Mayo Clinic Platform, and the bite was his way of making a point: “In healthcare, our data isn’t so good.”

The toenail

We both come out of registry work, so we asked about the gap between a general dataset and one built for a specific question. John’s example was a drug for an esoteric fungus, where you need to know the length of the patient’s toenail. “The registry will, the record won’t.”

Large language models now close much of that gap, because they can pull structured facts from notes. But that presupposes the fact was written down. “We always are careful not to be dogmatic,” he said. Leon’s version: you can’t judge the quality of data without first defining the purpose.

Cleaning comes first

Point these tools at a hospital’s raw records and you’ll find what John finds: 100 patients over 200 years old, and patients who’ve had three kidneys removed, thanks to an error copied from intern to fellow to attending. In his experience, the initial cleanup costs about a million dollars per academic health center and takes nine to 12 months.

Signal, then the trial

Leon raised John’s provocation that many questions may not need randomization. John’s example was Gelareh Zadeh, who chairs neurosurgery at Mayo. By his account, she pulled more than 8,000 glioblastoma patients before breakfast and compared those who died early with those who died late. A signal on glucose control has since gone into a randomized trial. “It is the gold standard,” John said, but a signal justifies running one. If an emulated trial shows clearly no signal, “maybe you don’t even wanna start the RCT.”

Leon called that triage. John’s reply was “yes, and”: early in COVID there was no time for an RCT, and he says that within two weeks, a look across 2,400 hospitals suggested ivermectin and hydroxychloroquine were “probably not a good idea.” Steve added that pharma has been doing this for years, including a look he worked on at Pfizer in the mid-2000s, where the data didn’t support the hypothesis. A no is useful too.

A thousand point solutions

“No healthcare organization, including Mayo, can buy one thousand point solutions,” John said. His hope is that agents will call qualified skills, each with a card describing its data, performance, and risk. Leon asked where to put the guardrails, since skills are unbounded and harms are not. Mayo uses a one-to-six risk scale, and John said it has more than one dimension.

Steve heard an old problem. MYCIN’s rules became too hard to maintain, and thousands of skills may be the same thing again. John didn’t claim otherwise. Mayo runs algorithms in passive mode first and requires surveillance afterward, because one can be “flawless in January and horrible by April.”

Too much trust

Our last question was what enthusiasts are overestimating. “I do not believe large language models are going to give us either AGI or trustable, repeatable clinical answers consistently,” John said. He expects a hybrid: Bayesian methods and knowledge graphs, with an LLM as the front end. “I think people trust LLMs too much.”

Listen to the full conversation.

You Might Also Enjoy