How our AI-text detector works
A plain description of the detector behind our arXiv measurement: what it is trained on, how it combines three signals, how we calibrate it to a low false-positive rate, and where it is weak. It is free, and we make no money from it.
Most AI-text detectors are easy to build and hard to trust. Two mistakes inflate most of the accuracy numbers you see. The first is data leakage, where the train and test split, or the feature vectorizer, can see test information, so in-sample numbers look excellent and then fall apart in the field. The second is a single-generator test set, where a detector tuned on clean ChatGPT output scores near-perfectly on paper and is useless on the mix of models people actually use.
Academic writing adds a third problem. The cost of a false positive is not the same as the cost of a miss. Flagging a genuine human paper as machine-written, where that label can affect a career, does far more harm than missing an AI-assisted one. So we judge the detector first on how often it is wrong about real human writing, and only after that on how much AI text it catches.
The data
The detector is trained on a mix of public labeled corpora, freshly collected human text, and synthetic AI text we generated ourselves, about 35,000 documents in total. The human side is deliberately pre-LLM where it matters: roughly 4,800 arXiv abstracts from 2019, 980 full-paper sections retrieved as clean prose, and reference-register writing from the 2019 revision of Wikipedia, frozen software documentation, and IETF RFCs. The AI side spans three generations on purpose: legacy generators through the MAGE and HC3 datasets, contemporary open and closed models (DeepSeek, Gemini, and Llama) as real text through the FAIDSet corpus, and current-generation output we generated ourselves by prompting recent OpenAI and Anthropic models to write in the exact registers we care about, such as paper sections matched to a real arXiv title and abstract.
Every document is assigned to train, validation, or test by a frozen hash of a source key, chosen so that a synthetic document and the human text it was seeded from always land in the same partition. That closes off the most common leak in this task, where a model is tested on an AI rewrite of a passage it already trained on.
Three signals, one verdict
The detector is a mixture of three experts that respond to different things, so they fail on different inputs.
- A neural expert: a small fine-tuned text classifier that reads meaning. Because it has a bounded context, a long paper is scored as the mean over several evenly spaced windows, so the score reflects the whole document.
- A lexical expert: a word-level model over word choice and phrasing. It uses word features only, which makes it close to invariant to markup and formatting.
- A stylometric expert: a 60-feature model over sentence-length rhythm, punctuation, and function-word habits, independent of topic.
Each expert's score is calibrated on held-out validation data, and the three calibrated scores, plus a few document-level features such as length, are combined by a small gradient-boosted classifier. The combiner is the only component that sees all three at once, and it never touches the raw text.
Calibration
We report on a held-out test set, with thresholds fixed on validation. The design target is the low-false-positive regime, so we report the full operating curve instead of a single number.
| Target false-positive rate | Academic-human false positives | Academic-AI recall | Long-form-AI recall |
|---|---|---|---|
| 0.2% | 0.4% | 83.9% | 88.9% |
| 0.4% | 0.4% | 85.0% | 88.9% |
| 1.0% (shipped) | 0.8% | 88.3% | 92.6% |
| 2.0% | 1.3% | 91.6% | 93.8% |
At a 0.4% false-positive rate on academic human text, that is 99.6% of genuine pre-2020 arXiv writing cleared, the detector recovers 85% of AI academic sections. Long-form and reference human text produce no false positives at any point in this table.
Where it is uneven
A single recall number hides the most important fact about any detector: it is not equally good against every model. At the shipped operating point it catches most families a large majority of the time, and Claude output under half the time.
Limitations
The detector is uneven across generators, as the figure shows, so any single headline recall is an average over our generator mix and will not transfer exactly to a different one. In deployment this lowers the flag rate: a model we cover poorly is one we under-flag.
We have not run a full paraphrase and adversarial evaluation of the current version. Lexical detectors are known to degrade under paraphrase, and while the neural and stylometric experts are meant to blunt that, we do not put a robustness number on this version until that evaluation is complete.
The held-out accuracy is a same-distribution number, and it overstates performance on genuinely novel text: a new generator, an unusual register, or another language. The detector is built and calibrated on prose-heavy scientific and reference English, and it is weakest where the text is least like that, including terse notation-dense writing such as pure mathematics, casual short text, and non-English text. Non-native-English scientific writing is a known false-positive risk.
Finally, a score is not authorship. The detector estimates whether text reads as machine-written, at a calibrated probability with a known error rate. It cannot separate a lightly-edited document from a wholly-generated one, and a single score is never grounds to accuse a specific person.
Try it
The detector is cheap to run and we make no money from it. You can try it for free on any arXiv paper here, and on your own text here.