| Risk | What the evidence shows | Planning use |
|---|---|---|
| Context-blind chatbot answers | Duke’s analysis of 11,000 conversations across 21 specialties found answers that are technically correct but medically inappropriate because they lack context. | Use a chatbot for background reading, then take your own details to a clinician. |
| Agreeable responses | Duke researcher Monica Agrawal notes that people like models that agree with them, so chatbots won’t necessarily push back. | Ask neutral questions. Do not treat agreement as confirmation. |
| Overreliance | Stanford’s Michelle Mello describes how easy it is to become complacent when tools work well most of the time. | Name a human reviewer for every AI output that feeds a care decision. |
| Errors that repeat | Mello says injuries can replicate over many patients. | Track outcomes across patient groups, not only case by case. |
| Thin vetting | Mello says there is much less testing and vetting than for drugs or medical devices. | Ask how a tool was validated before it enters a workflow. |
These statements come from Duke University School of Medicine and Stanford’s health policy group.
The dangers of AI in healthcare range from context-blind chatbot answers to biased models and weak data protection, and the risk is highest where nobody checks the output before it shapes care.
This article belongs to our wider coverage of digital health. It sets out what can go wrong, who it affects and what clinicians and patients can do about it.
Patient safety consequences
The most direct harm is a wrong or poorly fitted recommendation that someone acts on. The evidence points to three patterns: answers that miss context, tools that flatter the user, and failures that spread.
Answers that miss context
Duke researchers built the HealthChat-11K dataset from about 25,000 user messages in 11,000 conversations across 21 specialties. They found that chatbot answers can be technically correct yet medically inappropriate because they lack context. A documented example shows the problem. One chatbot correctly warned that a procedure should only be done by professionals, then went on to give step-by-step instructions for it. A warning followed by instructions can leave a patient with the impression that doing it alone is acceptable.
Agreeable tools and complacent users
Agrawal explains that chatbots are built to give answers people like, which means they may not push back on a leading or mistaken question. On the clinical side, Mello warns that it is very easy to become complacent when generative AI is right most of the time. A tool that is usually right makes the occasional miss harder to catch, because reviewers stop looking closely.
Failures that repeat at scale
A single clinician’s mistake usually affects one patient. A flawed model does not behave that way. Mello notes that AI injuries can replicate over many patients, and a 2019 Brookings analysis made the same point: an underlying problem in one AI system could injure thousands of patients. This is why monitoring has to look at patterns across a whole patient population, not only individual incidents.
Privacy and data exposure
Health data used with AI is not automatically private, and how private it stays depends on who holds it, what it is shared with and what the model can infer from it.
Brookings points out that AI can predict private information about patients even though the algorithm never received that information. Its example is Parkinson’s disease inferred from computer mouse tremor. Removing a diagnosis from a dataset therefore does not guarantee that the diagnosis cannot be recovered. Brookings also notes that lawsuits have been filed over data sharing between large health systems and AI developers, which shows that the terms of those arrangements are contested.
The route data takes matters. A common question is how does wearable data get to my doctor, because every hand-off between device, app, clinic and vendor is a place where data can be copied or reused. Sensitive categories such as mental health deserve the same scrutiny, and the checklist in Mental Health App Features: The 2026 Checklist For Secure, Effective Support – Bioconnected covers what to look for in an app.
- Ask about reuse. Find out whether your records or messages can be used to train a model, and who receives them.
- Share less with general chatbots. A consumer chatbot is not part of your care team, so give it only what the question needs.
- Clinicians and administrators should check contracts. Data-sharing terms with AI developers are where the disputes Brookings describes have arisen.
Bias and unequal care
Algorithmic bias affects patients by giving some groups recommendations that fit them less well, and it can make existing gaps in care wider.
Brookings explains the mechanism for data. A system trained mainly on records from academic medical centers will know less about, and so treat less effectively, patients from populations that rarely use those centers. It also warns that a model learning from health-system records could absorb past practice, such as suggesting lower painkiller doses to African-American patients. Researchers summarized by Rutgers add that such tools can generalize about patients of color and miss cultural context.
A concrete case comes from kidney care. Yale researchers describe how using race in kidney function estimates can lead to longer waits for Black patients to get on transplant lists. The transplant network responded by requiring kidney programs to assess their waitlists and reduce wait times. The same Yale work proposes practical steps against bias, including social determinants as alternative inputs to a race variable.
For clinicians, the practical reading is that a model should be checked on the patients actually in front of you, not only on the group it was built from. For patients, it is reasonable to ask whether a tool was evaluated on people like you.
Regulatory status and oversight
Oversight exists but is patchy, and a tool being in use does not mean it has been tested to the standard of a drug or device.
Stanford’s Alyce Adams says there are some 600 policies at state and federal level to regulate AI in health care. Mello, in the same discussion, says AI medical tools get much less testing and vetting than drugs or medical devices, and that most organizations lack robust vetting and governance. Large institutions are the exception: Stanford uses a process called FURM (Fair, Useful, and Reliable Models) to assess tools for fairness and reliability before use.
Brookings recommends collaborative oversight by both the FDA and other health-care actors, which fits what the evidence shows: no single body can see how a model behaves inside every hospital. The consequence for a smaller practice is that governance may fall to you. Without a formal process, a clinic still needs someone who owns the decision to adopt a tool and who reviews how it performs.
Why the failures happen
Machine learning models learn patterns from the data they are given, so weaknesses in that data and in testing carry directly into care.
Fragmented electronic health records
Brookings notes that health data is split across systems, providers and insurers, and that this fragmentation increases the risk of error and reduces how comprehensive datasets are. Electronic health records also take up a lot of clinicians’ time, which is part of why AI tools that promise to reduce that burden are attractive and why they are adopted quickly.
Tests that do not match real use
Duke’s team notes that most language models are tested on exam-style questions and answers. Real patient questions can be emotional, leading or risky, so strong exam performance says little about how a tool will handle them.
Human oversight in practice
Safeguards work best when a named person reviews the output and the patient understands what the tool is for.
For clinicians and health systems
- Assign a reviewer. Because complacency grows when a tool is usually right, make one person accountable for checking AI output that informs a decision.
- Vet before adopting. Ask how a tool was tested, on which patients, and whether it was assessed for fairness, as Stanford’s FURM process does.
- Watch outcomes across groups. Monitor results for different patient populations so a repeating error shows up early.
- Question the inputs. Where a variable such as race drives an estimate, check whether a better measure exists, as the Yale team suggests.
For patients
- Use chatbots as a first pass, not a final answer. Duke’s researchers advise checking any cited sources and relying on trusted sources for decisions.
- Give context deliberately. Missing context is the failure Duke documented, so bring your history and medicines to a clinician rather than relying on a general answer.
- Ask direct questions. Ask your care team whether AI was used in your care and how your data is handled.
Frequently asked questions
How many people ask AI chatbots health questions?
Duke reports that more than 230 million people do so each year, which is why the quality of those answers matters for public health and not only for individual users.
Does AI bias come only from training data?
No. A Yale study in PLOS Digital Health describes bias entering at four stages: training data, model development, model implementation and publication.
Why can a chatbot pass medical tests but still give poor advice?
Tests are usually built from exam-style questions and answers. Patients write differently, so a model can do well on an exam yet answer an emotional or leading question without the context a clinician would ask for.
Does AI-related harm affect only one patient at a time?
Not necessarily. Stanford’s Mello says AI injuries can replicate over many patients, so one flaw in a system can affect many people before it is noticed.
