Experts Warn of Risks from Relying on AI Chatbots for Medical Advice

As AI transforms healthcare, ensuring responsible integration remains a critical challenge.

Apr. 16, 2026 at 3:11am

Got story updates? Submit your updates here. ›

A ghostly, translucent X-ray photograph revealing the complex internal structure of a medical device or piece of healthcare technology, conceptually illustrating the challenges of implementing AI responsibly in the healthcare industry.As AI transforms healthcare, ensuring the responsible integration of this powerful technology remains a critical challenge.NYC Today

As the use of artificial intelligence (AI) expands in the healthcare industry, experts are raising concerns about the risks of patients relying on AI chatbots for medical advice. While proponents argue AI can boost productivity and affordability, critics caution that AI systems can provide inaccurate information without the accountability of a human doctor. The increasing adoption of AI in areas like drug discovery and medical billing also raises worries about rising healthcare costs. However, AI also shows promise in streamlining administrative tasks and aiding clinicians in diagnosis, though the long-term impact on the role of physicians remains uncertain.

Why it matters

The growing influence of AI in healthcare touches on critical issues of patient safety, medical ethics, and the future of the industry. Ensuring AI is implemented responsibly to enhance - not replace - human expertise is crucial as this technology becomes more prevalent.

The details

Proponents of AI in healthcare argue it can improve productivity and affordability, but critics warn of risks like patients receiving inaccurate medical advice from chatbots. Christabel Randolph of the Center for AI and Digital Policy says AI systems can "hallucinate" plausible-sounding but incorrect information, which could lead patients to delay care or take the wrong medication. The lack of accountability for AI-generated advice is also a major concern. While doctors face legal and professional consequences for bad advice, "when an AI gives you bad advice - who's responsible?" Randolph says. Some experts like the Heritage Foundation's Emma Waters worry AI will lead doctors to "outsource patient care" to algorithms. However, others like the Cato Institute's Jeffrey Singer believe AI could empower patients by giving them more control over their healthcare. The expansion of AI is also raising worries about rising costs, with research showing AI-driven billing practices may be inflating diagnoses without corresponding treatment.

  • In March 2023, an American Medical Association survey found 81% of physicians used AI, up from 38% in 2022.
  • In July 2025, the Centers for Medicare and Medicaid Services announced a new effort to create a digital health ecosystem with tech firms.
  • In March 2026, President Trump unveiled a national legislative framework and action plan for advancing AI in the U.S.

The players

Christabel Randolph

Associate director at the Center for AI and Digital Policy.

Emma Waters

Senior policy analyst at the Heritage Foundation.

Jeffrey Singer

Senior fellow at the Cato Institute and president emeritus of Valley Surgical Clinics Ltd.

Wayne Winegarden

Director of the Pacific Research Institute's Center for Medical Economics & Innovation.

Donald Trump

President of the United States.

Got photos? Submit your photos here. ›

What they’re saying

“The core risk is straightforward: AI systems can be confidently wrong. Large language models (LLMs), the underlying technology of 'AI chatbots,' are stochastic or probabilistic models. This means they generate responses based on randomness. Unlike a doctor who knows your history, examines you in person, and is legally and ethically accountable for what they tell you, an AI has none of those guardrails.”

— Christabel Randolph, Associate director, Center for AI and Digital Policy

“In the best case, AI integration in medical devices will help streamline some of the most cumbersome and administrative aspects of the job, especially paperwork, to free doctors up to spend more time with patients. There is also the very real opportunity to deep learning tools to expedite a doctors' ability to research and diagnose rare conditions that require large databases of information, cases, and analysis to complete.”

— Emma Waters, Senior policy analyst, Heritage Foundation

“I think it [the expansion of AI] could go a lot towards lowering health care costs, increasing affordability and expanding patient autonomy. Because right now, basically, patients are sort of at the mercy of the health care system, where they have to make an appointment with a doctor, and, of course, the medical profession is there's a lot of restricted entry points because of licensing and accreditation and all that.”

— Jeffrey Singer, Senior fellow, Cato Institute

What’s next

The Biden administration is expected to release new federal guidelines on the responsible development and deployment of AI in healthcare later this year.

The takeaway

As AI becomes more prevalent in healthcare, ensuring it is implemented responsibly to enhance rather than replace human expertise is crucial. Balancing the potential benefits of increased productivity and affordability with the risks of inaccurate advice and lack of accountability will be a key challenge going forward.