Of all the industries to embrace training in virtual reality, medicine has been one of the most eager to use a headset to learn. It makes total sense – training is often done in simulation centers, which require ample real estate and costly equipment, including lifelike mannequins. While they are realistic, they are also hard to scale – the limits of the physical space mean that only a certain number of people could be in there at any given time, and a student could only practice their skills a few times before they had to rotate out. This results in skill gaps which could then result in botched treatments and adverse outcomes for patients.
The current crop of VR simulations on the market have moved towards solving the problem, but gaps remain. For one thing, the quality of these simulations tends to be low – think floating heads and hands and blocky bodies. Producing this type of content at scale is a tradeoff – realistic avatars often cost a lot to create and are heavy, meaning they won’t look as good in widely available headsets. But when trainees see patients and low-res blobs and themselves as just disembodied limbs, it creates a sense of distance, and in an era when empathy and good doctor-patient relationships matter, this type of training can feel counterproductive.
These are just some of the principles that animated Gabo Arora when he launched Lightshed.Health in collaboration with the Johns Hopkins Schools of Nursing and Medicine. An experienced VR creator who directed the groundbreaking piece “Clouds Over Sidra,” for the United Nations, Arora taught for several semesters at Johns Hopkins and met members of the nursing faculty who shared their complaints about the current state of teaching with him.
“The core problem with current training methods is twofold: they are inefficient and ineffective,” Arora says. “Expert doctors at institutions like Hopkins are incredibly busy and often only available to teach procedures once a semester. When they are gone, students are often left with only low-quality learning materials, such as YouTube videos made by device companies or other students, which are clearly not working.”
He recognized that virtual reality could serve as a solution, but there were some issues early on. Live-action content, especially where precision is vital, suffered from “artifacting” when blown up, and performing intricate tasks within a visual “void” created matching problems. Arora and his team decided to pivot to mixed reality, utilizing newer, more intuitive headsets like the Quest 3 and its crisp pass-through video capabilities. Using holographic capture to place representations of the procedure directly into the student’s environment, the result feels like, as Arora puts it, “a textbook come to life.” The idea of the project is not to replace other forms of learning but to augment it at scale, allowing students to practice as many times as they need to before heading into a simulation lab.
Aside from the quality, the narrative element of the piece sets it apart from other content in the space. Students follow “Mr. Jones” from admission to resolution and encountered all aspects of his case, alongside family members who have both practical and emotional concerns with his treatment. Students not only practice hard skills, but soft skills as well – something that many programs don’t spend enough time teaching.
While incorporating AI was not part of the initial project, there are plans to do so in the future. AI could function as an “intelligent tutor,” customizing content based on student performance so certain subjects are emphasized while others, where the student might have more mastery, are deemphasized.
“The integration of mixed reality and AI in medical education represents a paradigm shift from passive learning to active, empathetic engagement,” says Dr. Harvey Castro, MD MBA, Emergency Medicine Physician and Healthcare AI Specialist. “When we combine AI-powered intelligent tutoring’s technical precision with mixed reality’s immersive empathy-building capabilities, we don’t just train technically competent physicians. We’re cultivating healers who understand both the science and the humanity of medicine. This technology doesn’t replace the human touch in healthcare; it amplifies our capacity for compassion at scale.”
While this project was built for the current line of mixed reality headsets, there is no reason that it couldn’t transition to smartglasses as future iterations of the technology are released. In ten years students could be learning almost entirely from smart glasses with hyper-realistic avatars and holograms, reshaping education and medicine as we know it. Until then, students can spend time with Mr. Jones and his family in mixed reality.