Comparing AI detectors: four things worth checking
Everyone advertises 95% plus. Those numbers are not comparable to each other. These four properties are, and they actually separate the tools.
Every detector on the market claims accuracy above 95%. None of those figures are comparable, because each was measured on a different test set, at a threshold the vendor chose, against generators the vendor picked. A number produced under conditions somebody selected for themselves is marketing, not evidence.
These four questions are checkable by you, on any tool, and they are the ones we ask.
1. Will it tell you how often it is wrong about human writing
The most useful question, and the one most tools quietly skip.
Accuracy blends two errors that have nothing in common. Missing generated text costs nobody anything that afternoon. Flagging something a person wrote arrives attached to their work. A tool can post a beautiful accuracy figure and still generate a large pile of wrong flags, because it runs at volume.
Turnitin publishes theirs: 4%.
We publish ours: 0.4%, which is 99.6% specificity, measured on 15,900 documents of real academic writing.
If a tool will not tell you this number at all, that is information too.
2. Does it show you where, or just how much
One document percentage hides the thing you need. A 30% spread evenly across every paragraph means something completely different from a 30% concentrated in one section, and the percentage alone cannot tell you which one you are looking at.
Per-passage output is worth more for the same reason a map beats an average. It also matters for reliability, because every detector is worse per sentence than per document. Turnitin's own numbers put the gap at roughly four times.
3. What kind of writing was it measured on
A detector measured on student essays and blog posts behaves differently on a methods section. Formal, regular, template-shaped prose sits closer to the boundary whoever wrote it, so a tool whose false positive rate came from casual text will exceed that rate on academic writing.
Ours was measured on academic prose, because that is what we built it for. On short casual text it is outside its best range and we would rather say so than pretend otherwise.
4. How does it cope with the document you actually have
Quick to test, and surprisingly often the deciding factor:
- LaTeX and markup. A lot of detectors read your preamble as prose and score it. See
- Length caps. Some stop well short of a full paper
- Language. Nearly all are English only. Ours says so explicitly instead of returning a
confident wrong answer
- Whether your text is stored. Worth knowing before pasting unpublished work
Why the advertised numbers vary so wildly
Three reasons, covered properly in why AI detectors disagree.
They tested different generators. Detection rates move by tens of points depending on which model wrote the text. Pick a favourable generator and you can advertise almost any number without saying a single false thing, which is what most published accuracy figures are.
They chose different thresholds. That choice alone moves a headline figure by tens of points.
They tested on data that resembles their training data, which flatters any classifier ever built.
The short version
- Ask for the false positive rate
- Check it reports per passage
- Check it was measured on writing like yours
- Run two tools. If they disagree wildly, the text is genuinely borderline, and that is worth
knowing before anyone acts on a single number
Ours is free and unlimited with no account. The full method is in how the unslop AI detector works.