The Scaffolding Problem: Why Raw AI Fails at Clinical Trial Matching

Most AI clinical trial matchers ask patients five or six questions — age, gender, disease, stage, grade — and return what they call a match. Adam Blum, a serial AI entrepreneur diagnosed with follicular lymphoma in August 2024, tried over a dozen of them. None returned an actual match.

“They try to broker you to pharma,” Blum told us. “But none of the matches that I was proposed were actually matches.”

The distinction Blum draws is between ranking and precision matching. Ranking systems score how relevant a trial seems to a patient’s profile. Precision matching checks whether the patient actually meets every eligibility criterion. The difference is concrete: patients who trust a ranking may show up to a trial site and discover they were never eligible. They’ve delayed their standard of care for nothing. The trialist has wasted time too.

What precision matching actually requires

For follicular lymphoma, Blum’s team identified 84 distinct eligibility attributes commonly used across trials. Each attribute requires multiple extraction questions — over 500 total. That’s the real complexity hiding behind clinical trial text.

The naive approach — handing trial text to an LLM and asking it to fill in the attributes — produces roughly 60% accuracy. “Little better than random,” Blum said.

CancerBot’s solution is a Prompt Workbench where biomedical subject matter experts iteratively refine extraction prompts for each disease. The prompts specify vocabulary mappings, flag common synonyms, and prevent the LLM from imputing values not present in the text. This gets accuracy above 90%.

The prompts can run to pages. But they’re built once per disease, not once per trial — a scalable tradeoff.

Making the logic tractable

Trial eligibility isn’t just a list of attributes. The criteria contain nested Boolean logic: AND/OR/NOT conditions that make human interpretation difficult and LLM reasoning unreliable.

CancerBot converts this logic into conjunctive normal form — a top-level checklist of AND conditions, with OR conditions pushed into named sets. Blum’s team found that every named OR set mapped to an existing medical term: CRAB criteria, SLIM criteria, renal insufficiency. The domain had already done the cognitive work of chunking these concepts into nouns. CancerBot just made the pattern explicit and computable.

The result is longer expressions that are paradoxically easier to read. A patient or clinician can look at a list and check items off rather than parsing nested conditionals.

Separating eligibility from goodness

Even after determining eligibility, there’s a second question: is this trial good for you? Blum’s grandmother also had lymphoma. She wanted low burden. Blum wanted maximum benefit regardless of burden. Same disease family, opposite priorities.

CancerBot asks patients what matters — risk, benefit, or burden — and ranks eligible trials accordingly. When information is missing, it tells patients exactly which questions to answer to complete the match rather than silently excluding them.

The broader pattern

CancerBot currently supports follicular lymphoma, multiple myeloma, and breast cancer, with CLL and mantle cell lymphoma next. The bottleneck is disease-specific prompt engineering — each cancer type requires its own attribute set and SME-refined prompts.

The lesson that extends beyond trial matching: raw LLMs fail at structured clinical tasks. What works is building careful scaffolding around them — vocabulary guardrails, logic normalization, iterative expert refinement. The AI is the engine. The engineering around it is the steering.

CancerBot is free and nonprofit. Learn more at cancerbot.org, or contact adam@cancerbot.org.

Listen to the full conversation on Practical AI in Healthcare.