A 10% edit cuts OpenAI’s watermark detection by 26 points
OpenAI’s EU-only text watermark drops from 92% to 66% detection after light editing, and misses short passages often.
By The Signal · · 5 min read
OpenAI's new watermark for AI-written text catches 92 percent of unedited passages at 400 words. Swap one word in ten for a synonym, and detection falls to 66 percent. Swap one in four, and it falls to 17 percent — roughly the odds of guessing right on a coin flip stacked against you twice over.
The company announced the system, called textGrain, on October 5, 2026, in response to the European Union's AI Act, whose transparency requirements took effect on August 2 that year. Within a few weeks, ChatGPT and Codex users inside the EU will have invisible watermarks applied to eligible text by default. API customers anywhere in the world can switch the feature on for select models starting now — but it stays off unless they ask.
A university lecturer in Warsaw is the test case nobody designed for
Picture someone grading essays at a public university in Poland. The EU AI Act exists partly so instructors like her have a machine-readable signal when a student submits AI-generated prose. OpenAI's watermark is meant to give her that signal. But the paper most likely to land on her desk — a 300-word response on a short-answer exam — is exactly the length textGrain handles worst. At 200 words, OpenAI's own figures show the detector flags only 80 percent of watermarked passages, against 95 percent at 400 words. A student who then runs the text through a thesaurus, swapping a tenth of the words, pushes detection down further, into coin-flip territory. The tool built to catch the exam answer does worst on exam answers.
The same gap applies to code. OpenAI's technical report notes that in math and programming output, word substitutions are more likely to break the underlying logic, so the watermarking algorithm has less room to operate without introducing errors. Codex, OpenAI's coding assistant, is one of the two products getting watermarked text in the EU — alongside a disclaimer, buried in the fine print, that the watermark "does not verify accuracy" and "does not identify the user."
Watermarking text is picking a different word on purpose, then counting how often
Large language models generate text one word at a time, each pick drawn from a probability distribution over plausible next words. An unwatermarked model simply takes the most likely candidates. textGrain periodically substitutes a less likely but roughly synonymous word, using a pattern tied to a secret key. Do this hundreds of times across a passage, and the aggregate word choices deviate from what an unbiased model would produce — a statistical fingerprint invisible to a human reader but visible to a detector that knows the key, the way a till roll shows a cashier's drawer ran $4 short even though no single transaction looks wrong.
The approach isn't new in concept. Google built SynthID-Text for this purpose, and Anthropic adopted it for Claude two months before OpenAI's announcement, applying watermarks globally rather than only where regulation requires it. Anthropic's rollout drew complaints from users who argued they supplied "the instructions, context, decisions" and that Claude was merely executing them — an objection to being tagged as co-author at all, independent of whether the tagging works. OpenAI, notably, had built a text watermark before this and held it back: The Wall Street Journal reported in 2024 that the company worried users would defect to competitors who didn't mark their output. The EU AI Act removed that competitive excuse, but only inside the EU.
The gap between "compliant" and "working" is the whole story
OpenAI is meeting the letter of a law that requires AI-generated text to be "identifiable" by machine — it is not required to be undefeatable, or even particularly reliable. The company's own technical report, co-written with researchers from the University of Pennsylvania and Yale, states a target error rate of 1 percent, then documents detection collapsing under conditions any determined user could produce with a word processor's synonym suggestions. A passage translated into another language and back, or run through a paraphrasing tool, degrades the signal further — OpenAI says it is still studying how badly.
Access to the detector itself is restricted to "approved researchers and expert organizations," not the public, and not the Polish lecturer grading exams. That restriction is partly an admission: OpenAI states plainly that the detector "produces both false positives and false negatives," and that a missing watermark "does not prove human authorship" — the text might be too short, too heavily edited, or generated by a different company's model entirely, since OpenAI's detector can't read Anthropic's or Google's marks and vice versa. Each company's watermark is keyed to its own system, so a universal detector doesn't exist and isn't planned by any single vendor.
OpenAI measured its own watermark and found it improved the writing
One detail in OpenAI's benchmarking data complicates the usual worry that watermarking degrades quality. Comparing watermarked and unwatermarked outputs across several evaluations, OpenAI found watermarked text scored slightly higher — even as the company maintains there's no meaningful overall performance difference. Nobody has explained why nudging word choice toward statistically rarer synonyms would help rather than hurt; it's the kind of result that invites a follow-up study OpenAI hasn't published yet.
The open question is whether a watermark this easy to strip is worth deploying at all, or whether it exists mainly to satisfy a regulatory checkbox while the harder problem — distinguishing AI assistance from AI authorship, across languages, across edits, across competitors' models — stays unsolved. OpenAI says it plans to open-source textGrain and keep refining it as "technology, standards, and evidence evolve." That's a reasonable position for a research team. It's a less reassuring one for anyone relying on the watermark today to settle a question of authorship that actually matters to them.
Does the watermark affect what ChatGPT writes for me? OpenAI reports no meaningful quality change, though benchmarks show watermarked text scoring marginally higher on some evaluations.
Can I check if text came from ChatGPT? Not yet — OpenAI's detector is limited to approved researchers and organizations, not the general public.