Can artificial intelligence rival doctors’ decision-making?
A new study by researchers at Cedars-Sinai, Tel Aviv University and digital health startup K Health says yes—at least in certain circumstances. The new study looked at the recommendations of K Health’s AI chatbot compared to those of the real-life doctors for patients who came to virtual urgent-care appointments with acute respiratory, urinary, vaginal, eye or dental symptoms. It found that the AI matched doctors’ clinical decisions in two-thirds of cases, and offered higher-quality care overall for the remaining one-third.
The researchers are publishing the results today in the peer-reviewed Annals of Internal Medicine, and presenting them at the American College of Physicians Internal Medicine Meeting in New Orleans this morning.
“If you look at places where the AI was better than the doctor, the AI was better at following guidelines,” Dr. Caroline Goldzweig, Cedars-Sinai Medical Network’s chief medical officer and an author of the study, told Forbes. “Where the AI wasn’t as good is where there was nuance.”
The retrospective study relied on data from Cedars-Sinai Connect, a virtual program that the Los Angeles-based academic health organization had rolled out using K Health’s assistant beginning in 2023. The chatbot converses with patients while simultaneously reviewing their medical records, then generates a ranked list of diagnoses and makes recommendations, such as for prescription drugs or lab tests. The study reviewed 461 visits by adults in June and July of 2024, and compared the AI’s top recommendation with that of doctors. To do that, the researchers had four outside physician adjudicators review the clinical decisions made by both human and machine after the fact.
In 68% of cases, the AI and physicians got equal quality scores for their clinical recommendations. In the remaining cases, the outside reviewers rated the AI decisions superior nearly twice as often, with the AI better in 21% of cases versus physicians’ 11%. The research found that the AI made potentially harmful recommendations 2.8% of the time, versus physicians’ 4.6%.
“I’m very proud of this. The AI is making better decisions and making fewer mistakes,” K Health cofounder and CEO Allon Bloch told Forbes, adding, “These are real-life cases of real-life people. Patients don’t bring clinical vignettes; they bring real-world symptoms.”
New York-based K Health, which launched in 2016, is one of a number of digital health startups that are trying to bring the technology into the doctor’s office. AI has the potential to transform doctor-patient interactions at a time when there’s a significant shortage of primary-care doctors and concern over how to provide care to the country’s aging population. At the same time, the volume of data, in both electronic medical records and research studies, has become overwhelming to doctors and can be easier for AI to ingest. Yet AI is only now beginning to be tested in real world situations, and concerns about it making mistakes run high. Even as AI models are validated and retrained with more data, there will never be 100% accuracy–just as no physician will ever be 100% accurate during their career.
Cedars-Sinai’s Goldzweig stressed that she did not foresee AI taking over from doctors, but that she hoped physicians would come to rely on it as a tool that could help free them up to spend more time talking with patients. The study’s results “tell me you can look at this AI and trust it, and you ought to take it into account when you are dealing with patients,” she said.
“For relatively straightforward problems, the AI might be great, and you might not need that much doctor intervention,” she added. “With complex patients with a lot of comorbidities, that’s where you really do need human intervention.”
Bloch, who had worked as a VC for Jerusalem Venture Partners and 83 North (formerly Greylock Israel), first started thinking about the challenges of data and medicine after his father experienced a stroke. Though Bloch had no previous experience in healthcare, he teamed up with Ran Shaul, the company’s chief product officer, and two others to start K Health in 2016. Eventually, they built out the New York-based company’s clinical-grade AI, which launched starting with acute care in 2019 and then expanded to full primary care.
With $384 million in venture funding from investors that include Claure Group and Valor Equity Partners, K Health is valued at $870 million. It has partnerships with Mayo Clinic, Hackensack Meridian Health and Hartford HealthCare, in addition to Cedars-Sinai, which is also an investor through its corporate VC arm. More than 10 million people have used its AI to date, independently or in conjunction with a doctor visit.
With the latest study results, Bloch said that he’d like to partner with top academic institutions to see if they should get FDA approval for certain capabilities and conditions. “I think it’s high time to think about what this all means for access, quality and cost,” he said.
MORE FROM FORBES
