Boris Berat is an operator, investor, and advisor. Co-founder & CTPO of Carna Health.
Some of the most useful challenges to my thinking about clinical AI come from conversations with my sister, Nevena Nemeš, a psychologist and experience strategist whose doctoral work explores human trust in AI. She asks a question technologists rarely frame precisely enough: Do we want people to trust AI more or trust it appropriately?
In healthcare, that distinction isn’t semantic. A patient who rejects a reliable recommendation may miss an opportunity for earlier care. A clinician who defers too readily to an unreliable recommendation can make an equally serious mistake. Both are failures of trust calibration.
Yet, many AI programs still treat trust as a sentiment to maximize. Leaders ask whether users trust the tool, whether adoption is increasing or whether explanations make people more comfortable. Those measures can reward the wrong behavior. In a high-stakes system, blind acceptance isn’t success, and skepticism isn’t always failure.
Building clinical software across countries has shown me how quickly a sound output loses value inside a human workflow. The same risk flag may be interpreted differently by a specialist, a primary care clinician, a community health worker and a patient at home. Trust isn’t a property of the model alone. It emerges from the relationship among the model, the interface, the workflow and the people expected to act.
Accuracy Is Only The Beginning
Research on automation describes poor trust calibration as a “core vulnerability.” A meta-analysis of 106 experiments found that human-AI systems often improved human performance but didn’t reliably outperform the better of the human or AI working alone. Better outcomes depend not only on model performance but also on whether people know when to rely on it.
Consider an AI-enabled screening system that identifies elevated chronic kidney disease risk using existing clinical, laboratory and point-of-care data. The model may be correct, but does the patient understand why follow-up matters when no symptoms are present? Can the clinician see which measurements drove the flag and whether the data is current? Is someone responsible for the next step?
If those questions are unanswered, model accuracy hasn’t become clinical value. The software produced an output, but the care system didn’t produce an outcome.
Four Conditions For Calibrated Trust
1. Make the evidence legible. Different users need different explanations. A specialist may need source data, thresholds and clinical rationale. A patient needs plain language explaining what the result means, what it doesn’t mean and what to do next. One generic “explainability” screen serves neither well.
2. Expose limits rather than hiding them. Users should see when data is incomplete, when the system is operating outside its validated context and when human review is required. Confidence without boundaries encourages automation bias. A system earns trust partly by making clear when it shouldn’t be trusted alone.
3. Preserve agency. People need a practical way to question an output, correct inaccurate information, request review or escalate a decision. In consequential settings, an explanation without recourse is incomplete.
4. Close the loop. A risk flag should lead to a named owner, a defined action and a timeframe. This is especially important when screening moves into pharmacies, community settings or the home. Extending access to testing creates little value if a concerning result disappears between systems or leaves the patient unsure what happens next.
Measure Reliance, Not Sentiment
A trust survey can be useful, but it can’t show whether trust is well calibrated. Healthcare leaders should also measure behavior: Do clinicians accept sound recommendations and challenge questionable ones? Do users recognize uncertainty? Do patients complete confirmatory testing and follow-up? Are errors or overrides concentrated in particular populations, locations or workflows?
These measures reveal whether people are using AI appropriately, not merely whether they like it. They also expose a problem that aggregate adoption rates conceal: The same interface may produce sensible reliance in one setting and dangerous overconfidence or rejection in another.
This is why psychology can’t be treated as a communications layer added after technical validation. It belongs in product requirements, acceptance testing and governance. Engineers, clinicians, behavioral scientists and designers should define expected human behavior alongside expected system behavior, including what should happen when inputs are missing, the recommendation conflicts with professional judgment or the user misunderstands the result.
Trust Is A Design Outcome
The next generation of clinical AI won’t succeed simply because models become more accurate. It will succeed when evidence, limitations, agency and accountability are designed into the entire path from prediction to action.
The goal isn’t to build systems people always trust. It’s to build systems that help people recognize when trust is warranted, when scrutiny is required and who remains accountable.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
