How the unslop AI detector works

If you don't care about the why, you can skip straight to the how.

AI detection is a controversial topic. There are two main reasons for this. The first is that AI detectors are being used as accusatory tools. A statistical score is presented as definite proof of AI authorship. The most visible offenders are schoolteachers and professors.

The second reason is complementary to this. The backlash swings too far in the opposite direction: "AI detectors don't work." This is also wrong. Detectors are statistical tools. Usually, they are right. Sometimes, they are wrong. And they are only useful in the hands of someone who understands their error rates and limitations.

Who is responsible for all of this? In my opinion, much of the blame falls on the detection companies. Their tools produce false positives and false negatives, but their outward appearance often hides that fact. Short and punchy sells better than long and elaborate.

What I wanted to do was build an AI detector that is easier to interpret while being more reliable than most. I settled on a design that minimises the false-positive rate. This allows users to be reasonably certain that when the detector score exceeds a certain threshold, the text is most likely AI-generated.

The sacrifice is that when the detector outputs a low score, we cannot be certain that the text was written by a human. It works best in one direction, and the direction I picked followed the logic of: "If you're going to make accusations, you might as well make the most solid type of accusation."

A false-positive rate of 0.4% is low enough that a strong positive result should warrant real suspicion, especially in a context where AI use is already common. But suspicion is not the same as certainty.

Consider a simple example: an AI model is instructed to reproduce a piece of human-written text verbatim. Is that text now AI-written? From the text alone, there is no way to know. The words contain no trace of whether they were typed by a person, copied by a model, or passed through several different systems before publication. A detector can provide strong statistical evidence about what a text resembles, but it cannot reconstruct the process that produced it.

What this detector is good for is work like the arXiv experiment I ran. You can use it to clearly show that there is a substantial upward trend in the publication of AI-written papers on arXiv. You may not be able to prove with certainty that a particular paper was AI-written, but across a sufficiently large sample, you can conclude beyond reasonable doubt that a substantial portion of these papers contains AI-generated writing and that this portion is increasing.

The dataset

I built my own text corpus out of pre-LLM academic writing and AI text derived from the human corpus. In total, it is made up of about 35,000 documents. It contains content from roughly 5,000 arXiv papers from 2019, 980 full-paper sections from other sources such as journal submissions, reference-register writing from the 2019 revision of Wikipedia, and IETF RFCs.

The AI part of the dataset includes texts from three different model generations: legacy generators such as GPT-3.5 through the MAGE and HC3 datasets, contemporary open and closed model outputs such as DeepSeek, Gemini, and Llama through the FAIDSet corpus, and current-generation output that I generated myself.

The self-generated data was by far the most interesting part of building the corpus. I used a process of extracting facts in the form of bullet points from the research papers and then built multiple AI-written versions. Each arXiv paper was used to generate one to four different AI versions. The goal was to get samples that are as close together as possible, separated only by being AI-written. Only the specific dynamics of word usage should matter, not the contents. The best thing about all this is that I was able to get it done right after the new model generation was released. This gives me a head start on other detectors that were trained on things such as the RAID dataset.

Before training a classifier on it, I assigned every document to the training, validation, or test set using a hash of a source key, chosen so that a synthetic document and the human text it was seeded from always land in the same set. That way, I avoid leakage.

The model

The detector I trained is a mixture of three experts that each respond to different text features.

1. The deep-learning part: a small fine-tuned text classifier that reads meaning. The goal was for it to be cheap enough that I could let anyone use the detector as much as needed while ensuring an acceptable baseline level of performance. For this reason, I chose e5-small. I started from a RAID-trained checkpoint and fine-tuned it on my own data.

1. The lexical part: the lexical expert is a term frequency, inverse document frequency model (TF-IDF) over word unigrams and bigrams, followed by logistic regression. It is much simpler than it sounds. We just look at word choice and short phrases, and this part learns the distribution of those expressions. I also removed all terms that occur in fewer than three documents, to avoid memorising things such as author names, obscure terminology, and paper-specific vocabulary.

1. The stylometric part: this one is the jankiest, I must admit. It uses 60 manually defined features and another logistic-regression model. The features describe things such as variation in sentence length, average word length, punctuation use, vocabulary diversity, and the frequency of common function words such as "the", "of", "and", and "to". The interesting differentiator here is variation in sentence length: LLMs tend to produce sentences of far more uniform length than human writers do.

Combining the experts

The three experts do not produce directly comparable scores. A 0.8 from the neural model does not mean the same thing as a 0.8 from the lexical model.

I calibrate each one separately on the validation set using isotonic regression. This turns the three arbitrary score scales into three roughly comparable probabilities.

The calibrated scores are then passed to a small gradient-boosted classifier together with three document-level features:

• document length;
• markup density;
• a language flag.

The final classifier is deliberately small. It only receives six inputs and uses trees with a maximum depth of three. There is not enough information here to justify a complicated model. A larger one would mostly become better at memorising the validation set.

The data separation is as follows:

• the three experts are trained on the training set;
• the combining model is trained on the validation set;
• all reported results come from the test set.

This distinction is one of the most important parts of the architecture. If the combining model were trained on the same documents as the experts, it could learn to trust scores that only look good because an expert memorised its training data.

Calibration

For academic text, I think the false-positive rate should come first. I therefore choose a tolerable false-positive rate and then measure how much AI text the detector still catches at that point.

The thresholds were fixed using the validation set. These numbers come from the held-out test set.

The target and measured false-positive rates are not identical because the target was chosen on validation and then measured again on unseen test data.

At the stricter 0.4% point, the detector clears 99.6% of genuine pre-LLM academic writing and catches 85% of the generated academic sections.

How much AI text the detector catches depends a great deal on which model wrote it.

Limitations

• The detector is built for prose-heavy scientific English. Short casual text, non-English text, and notation-heavy mathematics are outside its strongest area.
• A low score does not prove human authorship. It may mean that the text is human, or simply that the detector does not recognise the model or style that produced it.
• Performance varies by model. Most new models are caught reliably; however, Claude is currently something of a blind spot.
• Poor coverage mainly causes false negatives. A model the detector handles badly produces fewer flags. For the arXiv experiment, the measured share should therefore be interpreted as a lower bound.
• The test set is still related to the training distribution. Performance will be worse on a genuinely new generator, an unusual writing style, or another language.

Why it works well on academic writing

The detector was built around scientific English rather than the usual broad datasets. Its human data contains real academic writing from before widespread LLM use, and its generated examples are matched to that human source material.

It also scans several parts of long documents instead of classifying an entire paper from its first 256 tokens. The lexical model mostly ignores LaTeX and formatting, and the decision threshold is chosen around false positives rather than whichever threshold produces the nicest accuracy number.

None of this is groundbreaking. In combination, though, I am happy to say that my detector solves a very specific problem in the AI-detection space, and does so pretty well.