The associate professor at Columbia University’s School of Nursing had grown accustomed to having artificial intelligence tools help polish scientific papers for grammar, formatting, and other details. But a few weeks after submitting his latest research, the academic journal he was due to publish in came back with questions about a reference. The AI tool Topaz had used had silently inserted a fabricated source into his work.
“I felt deeply embarrassed,” Topaz, who leads a team at Columbia developing AI applications in healthcare, told Fortune.
“I’m an AI researcher. I know about hallucinations,” he said. “If this is happening to me, an AI expert, what happens to other people?”
That near-miss sent Topaz on an investigation to find out how often experts were getting subtly fooled by AI. The answer, it turns out, is a lot.
In a study published earlier this month in The Lancet, Topaz and his colleagues audited nearly 2.5 million biomedical papers and 97 million citations indexed on PubMed Central, the central repository used by clinicians and researchers worldwide. They found more than 4,000 fabricated references buried across nearly 3,000 papers. Not all the references were AI-generated, though Topaz said the steady rise in fake sourcing went “vertical” in 2024, shortly after AI tools in research entered more widespread use.
“It’s very reasonable that AI is highly associated with them now,” he said.
Over the past three years, the rate of fabricated references in biomedical literature has grown more than 12-fold. In 2023, one in 2,828 papers contained at least one fake reference, a rate that had risen to one in 458 by last year. Over the first seven weeks of 2026, the researchers found, one in 277 papers had at least one non-existent reference.
“I’m thinking this is just the tip of the iceberg,” Topaz said.
Hallucinations happen when an AI model prioritizes word patterns over accuracy. They are often harmless, but the stakes are different when AI errors begin infiltrating academic literature, as hallucinations risk undermining the scientific process.
Medicine is a field that builds on itself. Clinical trials cite earlier studies; systematic reviews then aggregate those trials, and medical guidelines finally cite those reviews. Doctors and nurses rely on those guidelines when they decide how to treat patients. A fabricated study planted at the start of that process doesn’t stay there.
“This is the evidence chain, that’s how we care for and treat people. If you put the fictional study at the bottom of the stack, the whole structure inherits it,” Topaz said.
“We’ve already seen paper mill articles included in systematic reviews informing clinical guidelines,” he added. “When a guideline paper cites a paper with a partially fictional references list, the evidence-based chain for treatment decisions is compromised.”
AI mistakes come for everyone
That AI is vulnerable to hallucinations has been known since ChatGPT first entered the scene four years ago, when students began to bravely submit specious AI-generated papers under their own name. But with a litany of tools, agents, and extensions now ubiquitous in nearly every profession, even experts in their field are getting tripped up by AI.
Take the case of Steven Rosenbaum. The author and filmmaker was in the headlines for all the wrong reasons this week after the New York Times identified a slew of inaccurate quotes throughout his new book, titled The Future of Truth: How AI Reshapes Reality.
The book carried blurbs from prominent journalists, including Nicholas Thompson, The Atlantic’s chief executive, and a foreword by Maria Ressa, the Nobel Peace Prize–winning reporter from the Philippines. It arrived, according to the Times, “to great fanfare.”
Rosenbaum’s book contained more than a half-dozen misattributed or entirely invented quotes, apparently generated by AI tools he had disclosed using in his acknowledgments. In a statement to the Times, Rosenbaum recognized the errors, calling the episode “a warning about the risks of AI-assisted research and verification.”
Instances like these might be inevitable given how widely AI is being used in expert-level knowledge work. Several journalism outlets, Fortune included, are now piloting the use of AI tools in reporting. Surveys suggest more than half of legal professionals are using AI tools to draft briefs and memos. A recent report by the American Medical Association found over 80% of physicians now use AI professionally to summarize research and prepare clinical documentation, a share that has more than doubled since 2023. Even Nobel laureates, such as Literature Prize winner Olga Tokarczuk, admit to using AI in their work.
As for research, one study last year by an American medical journal identified 36% of its papers contained at least some AI-generated text, although only 9% of researchers disclosed this when prompted prior to submitting their manuscripts. Another recent study found more than half of researchers are likely to be using AI tools while peer-reviewing other people’s work.
But as it turns out, experts in their field are no less likely to get duped. Topaz’s study of hallucinations in biomedical research joins a growing pile of anecdotes and datasets documenting embarrassing errors, including legal analyst Damien Charlotin’s catalog of 1,459 legal decisions citing AI-generated inaccurate content. Before he started the project a year ago, AI hallucinations in legal cases appeared two or three times a month. Now, there’s around five a day.
When experts get it wrong
Fake AI-generated research papers are already a problem in academia, increasingly difficult to parse through and threatening to overwhelm the peer-review system. But hallucinated references in real studies produced by humans could be just as widespread, and potentially even harder to track down.
The vast majority of papers tracked by Topaz contained only one or two fabricated citations, out of the several dozen references academic studies usually need to publish, suggesting most cases of AI hallucinations in research are unintentional.
But the publishing industry might not be prepared to handle the surging number of fake references, Topaz said. Verification methods differ between journals, and while some use software to check references and scan for AI-generated content, enforcement varies wildly. There is also no easy mechanism to retroactively screen the evidence chain to find original fake studies or references. So far, few journals have been able to identify hallucinations, as Topaz’s analysis found 98.4% of studies with fake references had not been retracted by publishers at the time of his audit.
It’s part of what people in the field have referred to as science’s “reproducibility crisis,” compounded in the age of AI by a rising flood of useless or unreliable AI-generated content that now permeates academic literature. But it’s a similar story in other fields that rely on output that can be reproduced. Stories in newspapers drive conversations and form the bedrock of future investigations. Legal decisions are eventually cited by lawyers and scholars in other cases.
Topaz said AI itself is not necessarily the villain, and he gladly uses it in his own work. “The problem is unverified AI output entering the permanent record,” he said. “The fix is not to stop using the tools, it’s to build verification into the workflow.”
“The longer we wait to put verifications in place, the harder it becomes to clean up,” he added.
AI hallucinations don’t care how well-versed in a subject users are. The mistakes are designed to look real, and they’re getting better at hiding. The more consequential the field—be it medicine, law, or journalism—the more dangerous errors become when they aren’t caught.
