GPs Cautioned: AI Notes Risk Dangerous Errors

The Growing Use of AI in Medical Documentation and the Risks Involved

As artificial intelligence (AI) becomes more integrated into healthcare, medical professionals are being urged to remain vigilant about the accuracy of AI-generated content. General practitioners (GPs) are increasingly using AI tools to assist with medical documentation, such as summarizing patient consultations. However, concerns have been raised about the potential for these tools to produce inaccurate or even fabricated information.

AI scribes, which automatically generate notes from conversations between doctors and patients, are becoming a common feature in many GP practices. These tools aim to reduce administrative burdens and improve efficiency. Yet, the Royal College of GPs has issued warnings that AI may misinterpret the subtle nuances of medical discussions, potentially leading to dangerous outcomes.

The Medicines and Healthcare products Regulatory Agency (MHRA) has also highlighted the risk of "hallucination" in AI systems—where the software generates false or misleading information. This issue is particularly concerning in medical contexts where precision is critical. The MHRA advises users to be aware of this risk and encourages manufacturers to implement measures to minimize harm.

To address these concerns, the MHRA has asked GPs to report any issues with AI scribes through its Yellow Card Scheme, typically used for reporting adverse reactions to medications. This includes instances of suspected inaccuracies, ensuring that any problems are documented and addressed promptly.

The British Medical Association’s GP Committee noted that the use of passive scribes has become widespread in general practice, with many clinics adopting standalone systems or integrating them with existing software. However, some healthcare professionals have expressed skepticism about the reliability of these tools.

Dr. Phil Whitaker, a UK GP who now practices in Canada, shared his experience with an AI tool that failed to accurately capture the details of his conversations. He reported that the system misinterpreted discussions about his move from the UK, recording incorrect information about patients relocating to Canada. Additionally, he found instances where the AI generated findings from examinations he had not performed and provided advice he had not given. While the company behind the tool advised users to review its output carefully, Dr. Whitaker found that the time spent correcting errors outweighed any productivity benefits.

A recent case highlighted by Fortune illustrates the potential risks: a London patient was mistakenly invited to a diabetic screening due to an AI-generated medical record falsely stating he had diabetes and suspected heart disease. Despite these incidents, the MHRA has not yet received any reports of adverse events directly linked to AI scribes in its database.

The government’s 10-Year Health Plan aims to accelerate the adoption of AI technologies, including AI scribes, by streamlining regulatory processes. A new national procurement platform will be launched next year to support GP practices and NHS trusts in adopting new technologies safely.

Professor Kamila Hawthorne, chair of the RCGP, emphasized the transformative potential of AI in healthcare but stressed the need for careful implementation. She called for close regulation to ensure patient safety and data security.

Public Concerns About AI in Healthcare

While the integration of AI into healthcare continues to grow, public confidence remains low. A recent poll revealed that fewer than one in three Britons are comfortable using AI features in the NHS App to diagnose their health issues. Among pensioners, the figure rises to 60%, with only 31% of respondents expressing comfort with the idea.

Health Secretary Wes Streeting announced plans to revamp the NHS App as part of Labour’s 10-Year Health Plan, aiming to provide every patient with a “doctor in their pocket.” However, critics argue that digitized services may leave behind those who are not digitally literate or prefer traditional care.

Helen Morgan, the Liberal Democrat’s health spokesperson, called for greater support for older adults and those less familiar with digital tools. She warned that without proper assistance, the push for AI-driven healthcare could exclude vulnerable groups.

Dennis Reed of Silver Voices, an organization advocating for elderly Britons, echoed these concerns, noting that many seniors may find the app difficult to use. He warned that reliance on AI could lead to exclusion from timely care.

Despite these challenges, the NHS and regulatory bodies continue to emphasize the importance of ensuring that AI tools meet strict safety and performance standards. The MHRA recommends that GPs only use registered medical devices and encourages the reporting of any adverse incidents through the Yellow Card Scheme.

In conclusion, while AI holds promise for improving healthcare efficiency, it also presents significant risks that must be carefully managed. As the technology evolves, ongoing vigilance, regulation, and user education will be essential to ensure that AI serves as a reliable and safe tool for both patients and healthcare professionals.