Adam Farren is the CEO of Canvas Medical, an EMR company accelerating everyday medicine.
Consider two hypothetical patients asking a consumer AI assistant like ChatGPT a health question. In the first one, a 58-year-old woman asks ChatGPT at 11:00 p.m. why she’s been short of breath climbing her stairs. She has her medical record connected, so ChatGPT Health can read her last three blood tests and notices her hemoglobin drifting from 13.4 to 9.6 g/dL. It recommends a doctor visit this week. The doctor’s workup finds iron deficiency; a colonoscopy finds the cause, and the tumor is stage I.
In the second scenario, a 67-year-old man using the free version without any health record information asks the same question. ChatGPT gives a careful, well-reasoned answer based on the limited context provided but doesn’t flag any urgency or next steps. He doesn’t see a doctor, and six weeks later, he’s in an emergency department with a heart attack that could have been avoided.
These scenarios are illustrative but point to something real. In this comparison, access to medical history looks like the clearest variable separating the two outcomes. Real-world results depend on more than that one factor, but a system’s access to a patient’s actual record is a meaningful, under-discussed piece of the picture.
As consumer AI assistants take on more of the work of interpreting sensitive health information, are they being built with the right incentives to prioritize accuracy over engagement? That’s a business-model question as much as it is a technology question.
Two Products, One Instructive Contrast
OpenAI’s recent product decisions show this tension playing out, as the company made two structurally similar bets that landed in different places.
OpenAI launched a personal finance experience on May 15 to Pro subscribers only, at $100 a month, connecting to more than 12,000 institutions through Plaid. It expanded to the $20 Plus tier a month later and has never touched the Free or ad-supported $8 Go tier.
Then, on July 23, OpenAI launched ChatGPT Health: sensitive data, expensive third-party connectivity, high stakes, a regulated adjacency—the same profile as finance in every structural respect. However, Health shipped everywhere at once, across Free, Go, Plus and Pro, to a far larger base. Three hundred million people now use ChatGPT weekly, up from 230 million in January.
Why The Business Model Is The Safety Question
Every business eventually aligns its product with how it gets paid. Google’s search got this right for two decades. Ads monetized best when they matched real intent, so a better index meant a better ad business. Facebook’s feed got it wrong: Revenue rewarded attention and time-in-app, so inflammatory content reliably outperformed true content on the metric that paid.
An ad-supported chat product sits closer to Facebook’s incentive structure. It’s rewarded for fast, fluent answers, not for surfacing what it doesn’t know. “I need four more pieces of information before I can answer that safely” is, on a free product’s metrics, a worse answer than one that just answers.
Think about that in health. Medicine is the practice of locating one specific patient inside a general population. For example, an A1C of 6.4 reads differently in someone newly prediabetic than in someone with 15 years of type 2 diabetes. A model without a patient’s chart can only answer for the population, and an ad-supported product isn’t structurally rewarded for closing that gap. Conversely, a subscription product implies that the subscriber evaluates what they pay for every month against whether it got the outputs right. Though incentives are just one factor influencing response quality—model capability, clinical safety policies and how a company chooses to invest its engineering resources—it’s fair to question whether that incentive translates into a better product.
This should be a major flag for healthcare and a no-brainer for why a paid subscription should be required, where the extra context is 100% needed with every query.
Attention Doesn’t Compound—Context Does
A maintained record gets richer with each visit, eventually giving a model something no population-level answer can: a patient’s own baseline and the ability to flag when something drifts. For example, a paying user would be more likely to follow up in chat to flag a wrong answer, clarify medication history, self-report if they followed advice and continue a thread with additional downstream symptoms or patient-reported outcomes. This type of data is gold for AI models to help them learn. The more a patient engages in conversation, the stronger the model’s memory becomes, which can lead to better answers.
Anthropic has already chosen that path, committing in February to keep Claude ad-free because ad incentives in a chat interface are nearly impossible to detect and tend to expand once absorbed into revenue targets. That’s an easier call for Anthropic as it is primarily a B2B business offering tools to the enterprise, and its consumer base is small and carries little cost. OpenAI doesn’t have that luxury: eMarketer (paywall) estimates it’s on pace to miss its own ad forecast by roughly 90% this year—under $1 billion against a $2.5 billion target—while reportedly still expecting ads to reach 36% of revenue by 2030. That’s real pressure to expand ad load into exactly the conversations (like health) where it should follow the subscription path.
Choosing The Harder Path On Purpose
Go back to the two patients. A smarter model won’t close the gap between the two outcomes. It’s closed by the unglamorous work of connecting patients’ full medical history and staying accountable to getting the answer right over years, not just once. That kind of work is difficult to fund with attention alone.
As consumer AI assistants take on a larger share of how patients get medical guidance, the business model deserves the same scrutiny as model capability and safety. The companies and regulators shaping this space should ask what standards, disclosures and incentive structures keep a product accountable. We’re setting defaults right now, industry-wide, for hundreds of millions of people who have no idea they’re being set. Getting the incentives right is as much a part of AI safety as getting the model right.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
