Are AI detectors accurate?

Not reliably enough to accuse anyone. OpenAI withdrew its own AI text classifier in July 2023 for its low accuracy: it caught 26% of AI-written text and wrongly flagged 9% of human text. Newer methods score better in their papers, but every threshold trades machine text caught against people flagged, as the scoreboard shows.

What is perplexity in AI detection?

How surprised a language model is by a text, on average. A perplexity of 15 means the model was as unsure as if it were picking from 15 equally likely words at each step. Text a model wrote tends to score low, because models pick likely words. GPTZero described it as how likely a model would be to choose the exact same words.

How much the surprise swings across a text. GPTZero described it as how much perplexity varies over a document: people mix plain stretches with odd turns, and a model is steadily predictable. On the stage, Carroll's memorized paragraph swings by 2.54 bits, under the 3-bit line, so adding burstiness to the rule doesn't save it.

Can AI detectors be fooled?

Yes, by making the text less predictable. Liang and colleagues found that asking ChatGPT to rewrite its essays in more literary language dropped detection from up to 100% to up to 13%. On the stage, the model's own text at temperature 2 with no top-p scores perplexity 1,418, far above anything Carroll wrote, and passes.

Can a detector prove a student used ChatGPT?

No. A verdict is a score crossing a line someone chose, and the line that catches more machine text also flags more people. Plain, formulaic or familiar writing scores as predictable whoever wrote it. OpenAI's classifier wrongly flagged 9% of human text before it was withdrawn.

Live model · AI detectors

How AI Detectors Actually Work

The classic AI detector asks how predictable a text is to a language model. Each word gets a surprise score, the average gives the perplexity, and low perplexity looks machine-written, because models pick likely words. But plain or familiar human writing is predictable too, so detectors flag people, and machine text written less predictably slips through.

Scoring Carroll's first paragraph

The detector reads the text with a language model, the word-pair counts from 6,929 words of Alice, and asks one thing of each of its 37 words: how surprised was the model to see it?

StopPerplexity detector

TextFlag perplexity belowRule

CH1 Predictable, over 1 in 4
CH2 Surprising, under 1 in 1,024

How to read the stage

The detector scores each word of Lewis Carroll's 1865 opening paragraph with the word-pair model from the other AI pages, trained on Alice's first three chapters. “to” after “beginning” gets a 90% chance, 0.1 bits of surprise; “book” after “the” gets 1 in 310, 8.3 bits. The average is 3.94 bits a word: perplexity 15. That's under the threshold of 20, so the detector calls Carroll's paragraph AI-like.

Why Carroll gets flagged

The model learned from that very paragraph, so every word is one it expects. Perplexity measures how familiar a text is to the model, not who wrote it. Real detectors fail the same way on famous texts: in 2023 Ars Technica reported that GPTZero called a section of the U.S. Constitution likely written entirely by AI, a text fed again and again into language models' training data.

No threshold gets it right

Run the same test on nine passages. The model's own text scores 6.9, 9.9 and 16; Carroll's paragraph, at 15, sits among them, so a threshold that catches all three flags Carroll too. A plain sentence from chapter 9, which the model never saw, scores 53 and gets flagged once the threshold reaches 100. At temperature 2 with no top-p, the model's text scores 1,418 and passes at every threshold.

Nine passages, one threshold

9 passages: 5 written by people, 4 by the model the detector measures with. Each gets the same test: flag it as AI if its perplexity is under 20.

StopScoreboard

Flag perplexity belowRule

Model text caught0 of 4People flagged0 of 5Flag below20

Average surprise per word, lowest first

flag below perplexity 20

The detector's record

Written by	Flagged AI	Passed
The model	0	0
People	0	0

Who gets flagged

Plain writing is predictable writing. Liang and colleagues ran seven detectors on 91 essays by non-native English speakers for the TOEFL exam: on average the detectors flagged 61% as AI, and 98% were flagged by at least one. Rewriting the essays with richer word choices cut the rate to 12%. Asking ChatGPT to use more literary language cut detection of its own essays from up to 100% to up to 13%.

What the model leaves out

Real detectors score text with large neural models, not word-pair counts from one book. GPTZero says it stopped using perplexity and burstiness in 2023 for a deep-learning model. Binoculars divides a model's perplexity by a cross-perplexity between two models, and reports catching over 90% of ChatGPT text at a 0.01% false-positive rate. What's modeled here is the original decision rule: low surprise, under a chosen line, means AI.

Questions

Are AI detectors accurate?: Not reliably enough to accuse anyone. OpenAI withdrew its own AI text classifier in July 2023 for its low accuracy: it caught 26% of AI-written text and wrongly flagged 9% of human text. Newer methods score better in their papers, but every threshold trades machine text caught against people flagged, as the scoreboard shows.
What is perplexity in AI detection?: How surprised a language model is by a text, on average. A perplexity of 15 means the model was as unsure as if it were picking from 15 equally likely words at each step. Text a model wrote tends to score low, because models pick likely words. GPTZero described it as how likely a model would be to choose the exact same words.
What is burstiness?: How much the surprise swings across a text. GPTZero described it as how much perplexity varies over a document: people mix plain stretches with odd turns, and a model is steadily predictable. On the stage, Carroll's memorized paragraph swings by 2.54 bits, under the 3-bit line, so adding burstiness to the rule doesn't save it.
Can AI detectors be fooled?: Yes, by making the text less predictable. Liang and colleagues found that asking ChatGPT to rewrite its essays in more literary language dropped detection from up to 100% to up to 13%. On the stage, the model's own text at temperature 2 with no top-p scores perplexity 1,418, far above anything Carroll wrote, and passes.
Can a detector prove a student used ChatGPT?: No. A verdict is a score crossing a line someone chose, and the line that catches more machine text also flags more people. Plain, formulaic or familiar writing scores as predictable whoever wrote it. OpenAI's classifier wrongly flagged 9% of human text before it was withdrawn.