In the development that can reshape how hospitals deploy AI, free and open AI systems match the performance of leading proprietary tools to address complex medical cases that often bother human doctors.
Researchers at Harvard Medical School found that Llama 3.1 405B is an open source AI model whose code is publicly available and executed at the same level as Tech Giant OpenAI’s flagship closed freight model.
When 92 clinical protocols with challenging diagnosis were tested in the New England Journal of Medicine, open source challengers correctly diagnosed 70% of cases, compared with 64% of cases of GPT-4. Even more impressive, Llama listed the correct diagnosis as 41% of its first recommendation, slightly better than GPT-4’s 37%.
“To our knowledge, this is the first time that the open source AI model matches GPT-4’s performance in challenging cases of physician evaluation,” said Arjun Manrai, assistant professor of biomedical informatics at the Blavatnik Institute at Harvard University. “It’s amazing how llama models catch up so quickly with the leading proprietary model. Patients, care providers and hospitals will benefit from this competition.”
These findings may mark a turning point in medical AI, as hospitals believe that the AI system should adopt. Open source models such as Llama offer several advantages – they can run locally on hospital servers, keeping their internal sensitive patient data rather than sending it to external servers operated by commercial entities.
“The open source model may be more attractive to many CIOs, hospital administrators and doctors because it’s simply different to take data out of the hospital and heading to another entity, or even a trustworthy person.”
Both AI systems in the study rely on similar approaches – training on large numbers of data sets, including medical textbooks, research papers, and anonymous patient information. When new clinical situations arise, they compare them with training data to propose possible diagnosis.
The researchers carefully controlled any advantages that Llama might be exposed to certain test cases during training. These include 22 new cases released after the end of the training period for Llama, and the open source model performed better in these new cases, with a correct diagnosis of 73%, and a list of the correct answers as its best recommendations, 45% of which time.
Another major advantage of open source models is their flexibility. “That’s the key,” Barkley noted. “You can fine-tune these models in a basic way or in a refined way using local data so that they adapt to the needs of your own physicians, researchers and patients.”
The gap between open and closed AI systems is similar to early technological shifts, for example, when open source electronic health record systems start to challenge proprietary platforms. While closed developers such as OpenAI provide customer support and hosting, the open source model requires institutions to handle setup and maintain themselves.
“As a doctor, I’ve seen a lot of attention to powerful big language models around proprietary models that we can’t run locally,” said Adam Rodman, assistant professor of medicine and researcher at HMS ISRAEL DEACONECESS Medical Center. “Our research shows that open source models may be just as powerful, giving physicians and health systems more control over how these technologies are used.”
The bet to improve diagnostic accuracy is high. According to a 2023 report cited in the study, approximately 795,000 patients in the United States die or have permanent disabilities each year due to diagnosis errors. In addition to the loss of a person, delayed or incorrect diagnosis increases medical expenses through unnecessary testing and treatment.
“In the current health infrastructure, using AI tools wisely and responsibly can be valuable co-pilots for busy clinicians and act as trusted diagnostic assistants to improve the accuracy and speed of diagnosis,” Manrai said. “But it is crucial that doctors help push these efforts to ensure AI is useful to them.”
As hospitals increasingly evaluate AI clinical use tools, the competitiveness of open source options may lead to more affordable and customizable solutions. For patients, this may eventually translate into a more accurate diagnosis without privacy concerns that send data to third-party companies.
Given the impressive diagnostic capabilities of open and closed systems, choices in healthcare facilities may increasingly depend on factors beyond original performance, including data privacy, custom requirements and implementation costs.
If you find this piece useful, consider supporting our work with a one-time or monthly donation. Your contribution allows us to continue to bring you accurate, thought-provoking scientific and medical news that you can trust. Independent reporting requires time, effort, and resources, and your support makes it possible for us to continue exploring stories that are important to you. Together, we can ensure that important discoveries and developments attract those who need them the most.
