Calibrated to Nothing
*The damning thing about AI detectors isn't that they're wrong — it's who they're wrong about, and the fact that the only audit of the ecosystem is the ecosystem.*
The occasion: a post on a DePaul blog late last month — "The Limitations of AI Detectors: What They Get Wrong About Human and AI Texts" — walking through the now-familiar failure mode: professional human writers, writing professionally, getting flagged as machines. It's a student blog post, not a study, and that's part of the story, but hold that. The pattern it documents is real and widely repeated.
Grant the tools their strongest version first: this is a genuinely hard problem. Detectors are asked to separate human text from machine text at exactly the moment the two are converging by design — the machines are trained to sound like us, and we're all being coached to sound legible. Every measurement instrument has an error rate. The vendors even publish their limitations, which is more candor than most of edtech offers. So the charge here is not "the tools are sometimes wrong." Every tool is sometimes wrong.
The charge is that the errors are *differentially* wrong in a way that tracks who the writer is. The literature around these tools reports that non-native-speaker prose gets flagged as machine-generated at rates that would end any other instrument's career, while native-speaker prose largely passes. And I'll note the honest complication myself: that figure circulates as a free-floating number — repeated across explainer posts from Grammarly, Editage, and Trinka — often without the methodology attached. That's not a reason to doubt the harm; it's a second instance of it. An instrument whose errors cluster on non-native speakers isn't a flawed instrument. It's a discriminatory one wearing the costume of neutral measurement — and it's deployed precisely against the people with the least standing to fight a score: students, freelance writers, job applicants.
A broken instrument is a nuisance. A broken instrument whose errors cluster on one population, deployed against that population, is policy.
But the cleanest argument against these tools isn't the differential error rate, which requires trusting numbers whose provenance is murky. It's the pre-generative-AI false flag, and nobody gives it enough weight. If a document that provably predates the technology — written before any generative model existed — comes back with a high machine-generated confidence score, that is not an error. There was nothing to detect. An error rate implies the instrument has a true zero somewhere, a case where it correctly says "human." A flag on text from before the technology existed shows the confidence score isn't measuring anything at all. It's calibrated to nothing. That's a different category of failure, and it's the one that matters, because "unreliable" and "meaningless" get treated very differently anywhere evidence is taken seriously.
Which brings us to the closed loop. Who produces the limitations literature? The ecosystem itself. GPTZero — a detector vendor — publishes the explainer on why your writing got flagged. Grammarly, which sells AI writing features, explains how detectors work. The vendors write the caveats about the tools the vendors sell, the error rates circulate as numbers detached from methodology, and the only accountability mechanism that has ever actually produced results is a lawsuit. That's not an audit structure. That's a hall of mirrors with an invoicing department.
So here's the prediction, offered plainly: the argument that eventually kills these tools in court will not be the error rate, and it will not be the differential harm, though both will be in the complaint. It will be the pre-AI document — because it converts "unreliable" into "provably meaningless," and meaningless instruments don't get a warning label. They get excluded.