This is insane. AI makes things up. Lawyers got caught. Now it's in your emergency room.
This writer went to urgent care with a vein bulging out of the side of the head, running from the forehead into the temple. Urgent care sent the writer to the emergency room.
At the ER, the staff asked if it was okay to use AI. The writer said yes. Then the doctor moved fast, working down what looked like a computer checklist. And the questions kept coming back to one thing: headaches. How bad are your headaches? How often? The writer was confused, and said so: "I don't really get headaches." The staff looked confused back.
Then came the MRI. Then the CT scan. Everything came back "clear." Clear of what? Of the headache problem nobody had. The vein the writer came in for never got explained. They were chasing the wrong dog, and the writer walked out with no answer and no plan.
Maybe that was just a rushed doctor. Nobody can say for sure what the AI suggested. But it's exactly the kind of mistake AI is famous for: grab onto the wrong idea, then follow it with total confidence. And AI is now inside a huge share of American medicine. This article lays out the proof, piece by piece, of why that should worry everyone.
First: AI makes things up, and sounds sure about it
The AI tools in the news, like ChatGPT, don't look facts up the way you'd look in a dictionary. They predict what words usually come next, based on mountains of text they've read. Most of the time that sounds right. But when they don't know, they often don't say "I don't know." They invent an answer that sounds right, with names, numbers and details, and they say it just as confidently as the truth.
Experts call this a hallucination. A federal judge put it simply: AI "hallucinates" when it "generates facts that, well, just aren't true" (U.S. Court of Appeals, Fifth Circuit, quoting Judge Kevin Newsom).
The proof: AI got caught lying in court, over 2,000 times
Courts are the perfect test, because every fact in a legal filing can be checked. And AI keeps failing it.
- The first big one, 2023: two New York lawyers filed a brief full of court cases that did not exist. ChatGPT had invented them, along with fake quotes. When the judge asked, they kept standing by them. The judge called one fake ruling's reasoning "gibberish" and fined them $5,000 (Mata v. Avianca, Opinion and Order on Sanctions).
- It didn't stop there. A researcher in Paris tracks court decisions that found AI-invented material. As of today, the count is 2,231 cases around the world, more than 1,500 of them in the U.S. In July 2025 it was 206 (AI Hallucination Cases database, NPR via GPB).
- Big, expensive law firms too: two firms were fined $31,100 for what a judge called a "collective debacle" of "bogus AI research" (ABA Journal). Lawyers defending Alabama's prisons were thrown off a case over ChatGPT-made citations (report). Even the U.S. Justice Department was called out in 2026 for citing a case that doesn't exist (Bloomberg Law).
- Even judges: in 2025, two federal judges had to withdraw or fix rulings with made-up quotes and wrong facts after staff used ChatGPT and Perplexity (FedScoop).
- Even an AI expert: a Stanford expert on misinformation filed sworn testimony with citations to articles that don't exist. The judge said it "shatters his credibility with this Court" (Stanford Daily, Wikipedia).
And the professional tools are not safe either. Stanford researchers tested general chatbots on legal questions: they got things wrong 58% to 88% of the time, depending on the model. Then they tested the expensive, specialized legal AI tools sold to law firms, the ones advertised as reliable. Those still made things up 17% to 33% of the time (Stanford, Journal of Legal Analysis, Stanford, "Hallucination-Free?").
Think about that: lawyers are trained to check every citation, and thousands of them still got fooled. The researchers also found these tools tend "to assume the veracity of a premise even when false." In plain words: if you start with the wrong idea, the AI goes along with it. Like a headache that wasn't there.
Now the same technology is in your doctor's office, and your ER
- 81% of U.S. doctors now use AI in their practice, up from 38% in 2023, according to the American Medical Association's 2026 survey. About 1 in 6 use it to help with diagnosis (AMA, AMA survey details).
- 61% of emergency doctors surveyed use AI tools in patient care. Among AI tools they used that their hospitals didn't provide, the most common was ChatGPT, at 63% (JACEP Open survey, via Medical Dialogues).
- About 63% of U.S. hospitals using Epic, the most common hospital records system, had adopted or were rolling out AI that listens to your visit and writes the doctor's notes, as of June 2025 (American Journal of Managed Care).
Medical AI has already been caught making things up
- It invented words in medical records. A popular speech-to-text AI used to transcribe doctor visits sometimes made up whole sentences nobody said, including "imagined medical treatments," the Associated Press reported. A Cornell-led study found about 1 in 100 recordings got invented phrases. One medical tool built on it was used in about 7 million visits, and it erased the original audio, so nobody could check (Cornell, Associated Press, via Anchorage Daily News). "Artificial intelligence is making up something from nothing," said researcher Allison Koenecke.
- Google's medical AI invented a body part. In its own research write-up, Google's Med-Gemini described damage in the "basilar ganglia." There's no such thing. It mashed together two real parts of the brain. Google quietly called it a typo (The Verge, futurezone).
- A hospital sepsis alert missed two out of three cases. When University of Michigan doctors checked a widely used sepsis-warning model, it missed 67% of patients who actually had sepsis, while sounding alarms on nearly 1 in 5 hospital stays (JAMA Internal Medicine).
- Chatbots run with fake medical details. Mount Sinai researchers slipped one made-up detail into 300 patient cases. Six chatbots repeated or built on the fake detail 50% to 83% of the time. "A single misleading phrase can prompt a confident yet entirely wrong answer," the researchers warned (Medical Economics).
- Real people using chatbots did no better. In an Oxford study of 1,298 people, chatbots alone could name the right condition about 95% of the time. But when real people used them to figure out what was wrong, they got it right less than 35% of the time, no better than people who didn't use AI at all. The authors' verdict: today's chatbots "are not ready for deployment in direct patient care" (Nature Medicine, via Nature Asia).
The patient-safety watchdogs agree. ECRI, a leading independent patient-safety organization, named AI the No. 1 health technology hazard of 2025, and misuse of AI chatbots the No. 1 hazard of 2026. In its testing, chatbots suggested wrong diagnoses and unnecessary tests, and even invented body parts (ECRI 2025, ECRI 2026).
The scariest part: smart people follow the machine, even when it's wrong
There's a name for this: automation bias. It means trusting the computer's answer over your own judgment, even when the computer is wrong. It's like following your GPS into a lake because the screen said "turn right."
Doctors are not immune. They're trained in medicine, not in catching a computer's mistakes:
- Radiologists: when researchers fed radiologists deliberately wrong AI suggestions on mammograms, less experienced readers' accuracy fell from about 80% to under 20%. Even experienced radiologists dropped from 82% to about 46% (RSNA, on the study in Radiology).
- Doctors and nurses: in a study of 457 clinicians, a flawed AI pulled their diagnostic accuracy down by more than 11 points. Showing them the AI's "reasoning" didn't fix it (JAMA).
Put it together, and the danger is obvious
Follow the logic, one step at a time:
- AI makes things up, and sounds sure when it does. That's proven, in court, thousands of times.
- AI grabs onto whatever idea it's given first, even a wrong one, and builds on it.
- Most doctors now use AI, and many ERs have it built right into how they work.
- ERs are the most rushed places in medicine, where there's the least time to double-check.
- People, including experts, follow the computer even when it's wrong.
- Saying "yes" to AI doesn't tell you what it's doing. Many hospitals now ask permission, like this writer's ER did. But nobody explains which questions the AI is steering, or what it might have gotten wrong, so patients don't know when to push back.
A machine that invents things, used by busy people who trust it, on patients who can't see what it's doing. Lawyers had a judge catch their AI's mistakes. In an ER, the person who catches the mistake might be you, or nobody.
