The Efficiency of the 'Old School': Why Basic Statistics Outperform AI Detectors
AI-generated image · Bay Street Wire
While the industry chases complex LLM-based classifiers, a practitioner's experiment reveals that simple machine learning models can spot AI patterns with surprising accuracy.
In the current arms race to identify AI-generated content, the industry has leaned heavily into over-engineered solutions. We are told that only an LLM can catch an LLM. But as I often argue in this column, the most sophisticated tool is rarely the most effective one. The real story isn't the emergence of a 'perfect' detector; it is the fact that basic statistical patterns remain glaringly obvious to anyone using the right, albeit simple, tools.
As Hacker News reported, a technical analysis by lyc8503 suggests that the 'ghost' of perfect detection is being chased through the wrong channels. The author's research indicates that mainstream LLM-generated text exhibits strong statistical patterns that can be effectively distinguished from human writing using traditional machine learning models. This is a critical distinction: we aren't talking about neural networks with billions of parameters, but 'classical' machine learning.
**The Failure of Perplexity**
One of the most touted methods for AI detection is text perplexity. As lyc8503 describes it, the technique calls for an existing LLM to estimate the probability of each word in a sentence. If most words score highly in the model’s predictions, the text is flagged as AI-generated; if the word choices are unpredictable, the text is deemed more likely human-written.
However, lyc8503 found this approach to be fundamentally flawed in practice. The results were plagued by false positives and false negatives, with no reasonable threshold for accuracy. Beyond the statistical noise, the operational costs are prohibitive. Lyc8503 notes high inference costs, poor generalization across different models, and the difficulty of deploying large models locally or integrating closed-weight models. In short, the modern approach lacks both elegance and reliability.
**The Power of 'Old-School Alchemy'**
When the high-tech options failed, lyc8503 turned to what they describe as 'old-school alchemy' using the scikit-learn library. Rather than relying on the probability distributions of a live LLM, the researcher utilized Linear SVC (Support Vector Classification) and Naive Bayes—classification tasks that are staples of basic machine learning.
To test this, lyc8503 constructed a dataset from texts scraped in 2023 from platforms described as 'Ford-like' and 'River-like,' filtering for nearly 10,000 multi-thousand-character pieces published between 2010 and 2022 (pre-dating the release of ChatGPT). To create a matching set of AI content, the researcher used an LLM to generate summaries of these human texts and then had the LLM regenerate full articles based on those summaries.
To keep costs low, lyc8503 utilized several API channels, including Gemini (via CLIProxyAPI), Qwen (via qwen-code), GLM-5 (via OpenRouter), and promotional plans for Kimi, Deepseek, Doubao, and GLM-4.7. The resulting model, which used gemini-3-flash for summaries and seven other models for generation, achieved a single-sentence detection accuracy of approximately 85% on the test set.
**Analysis: The Signal in the Noise**
*Opinion: This is where the industry is getting it wrong. The belief that we need 'AI to catch AI' is a marketing narrative, not a technical necessity. What lyc8503's experiment proves is that LLMs, regardless of their sophistication, have detectable word-choice patterns. These patterns are not hidden; they are systemic. When a Naive Bayes classifier—one of the simplest algorithms in the data scientist's toolkit—can pick up a signal this strong, it suggests that AI-generated text isn't just 'different' from human text; it is statistically repetitive in ways that are trivial to map.*
Lyc8503 suspects that many commercial 'AI plagiarism checkers' are actually operating this way under the hood, utilizing simple classifiers rather than the complex, resource-heavy LLM-detectors they market to the public.
The takeaway for practitioners is clear: stop looking for a magic bullet in the form of a larger model. The most effective way to identify synthetic content is not to mimic the AI's intelligence, but to expose its predictability. As lyc8503's work demonstrates, the most reliable signals aren't found in the 'intelligence' of the text, but in the boring, predictable statistics of its construction.