A chatbot hallucination nearly sent US troops onto a Chinese ship

A US Special Operations Command analyst fed intelligence about a Chinese ship's manifest through a chatbot this spring, and the report that came out said the vessel was carrying parts for a nuclear weapons programme. CNN, citing four sources familiar with the episode, reports that aircraft were already in the air and armed personnel were ready to board before officials looked harder and found the assessment had been built with artificial intelligence, and that the chatbot had misidentified what the ship was carrying. One source told CNN the near-miss almost started a war.

It was caught just in time

The operation was stopped at the last stage of planning, not by a correction from the ship's operators but by officials deciding to check the working underneath the report. That is the whole margin here: one more round of scrutiny before people with weapons arrived on the deck of another country's vessel. According to CNN, the chatbot had fused open-source intelligence with secret signals intelligence held by the US government and produced a single confident-sounding document from both. Nothing in the report flagged which claim came from where, or that a language model had done the fusing.

This is what hallucination looks like at scale

Most AI hallucination stories end with an embarrassing paragraph in a newsletter or a wrong answer in a customer chat. This one is different in kind, because the model was sitting inside a workflow where its output was treated as intelligence rather than as a draft. Language models generate fluent, well-structured text whether or not the underlying claim is true, and an analyst under time pressure reading a tidy summary gets no signal at all about which sentences are grounded. The failure was not that the chatbot made something up. It is that nobody had a reason to doubt it until the boarding party was already on standby.

The Pentagon is pushing AI into exactly this work

The episode lands in the middle of a deliberate expansion of AI in military decision making. The Department of Defense rolled out an AI acceleration strategy in January that aims to make appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component. Separately, Bloomberg reported that a Pentagon investigation into the missile strike that killed schoolchildren in Iran on 28 February identified failures including an overreliance on artificial-intelligence technology. Two incidents, two very different outcomes, one shared root cause.

Our opinion

The honest lesson here is not that AI cannot be used in intelligence work. Analysts have been drowning in material for decades and machine assistance is genuinely useful. The lesson is that a system which produces confident prose must never be the last step before an armed decision. What stopped this operation was a human choosing to re-read the source material, which is precisely the diligence that software is supposed to make easier and instead quietly discouraged. Any deployment inside a command chain needs provenance attached to every sentence, an obvious marker of what a model inferred rather than retrieved, and a rule that machine-generated assessments cannot be the sole basis for action. The alternative is what nearly happened in the spring: a war that began because a chatbot guessed, fluently, that the manifest said something it did not.