Fine-tuned AI-text detector
Veritarach
A binary text classifier fine-tuned from microsoft/deberta-v3-base, trained on a dataset I built, deployed behind HTTPS on a real server, and registered on a decentralised inference protocol. Then I built a separate harness to check whether the headline number actually meant anything.
Live and serving, registered as an active node
- rows in the training set
- 96,646
- F1 on its own held-out split
- 99.86%
- spent on rented GPU training
- $1.20
The problem
Public AI-text detection datasets skew heavily toward one generator. HC3 pairs human answers with ChatGPT answers, which is a good starting point and a narrow one, because a classifier trained on it learns the habits of a single model family rather than the general shape of generated text.
So the dataset had to be built rather than downloaded. HC3 for the paired human and AI examples, Wikipedia for human writing that is not answer-shaped, and around 593 samples I generated myself across three different providers to cover styles the base data missed.
How it works
The data pipeline handles fetching, generation and splitting. Sample generation tracks spend against a budget and writes a manifest, so an interrupted run resumes instead of restarting and re-billing.
Splitting uses a group-aware strategy from scikit-learn, because HC3's human and AI answers come in pairs. A naive random split puts one half of a pair in train and the other in test, and the resulting score measures leakage rather than learning.
Training ran on rented GPUs, two full runs, under $1.20 in total compute.
Serving is a FastAPI application in a Docker container on a DigitalOcean droplet, with Caddy terminating TLS and pulling certificates from Let's Encrypt automatically.
Registration on the protocol meant setting up an EVM wallet, funding it on a testnet, and submitting an on-chain transaction to activate the service as a node for the text-detection intent.
Where it landed
The model reports 99.86% F1 on its own held-out test set, and the deployed service is registered and active on the protocol. I verified the registration by querying the protocol's own backend and checking the wallet's transaction nonce, rather than trusting the success message the tooling returned.
Then I built Veracia, a separate evaluation harness with an independently constructed holdout, and pointed it at the live deployment. It found that the score does not survive contact with text outside the training distribution: recall dropped to 0.042, catching one of 24 AI samples, and the 23 misses were returned as human with 0.98 or higher confidence. Across a wider cross-model set, 38 of 100 clearly-AI samples came back as human.
The model had learned a narrower rule than the metric implied, roughly "call it human unless it looks like the training data". That is a real result about the project, and it is the reason the F1 figure on this page is always stated against its own split.
See for yourself
What the model actually says
Six passages, three written by a person and three by a language model. Pick one to see the response the deployed service returned for it.
honestly i spent like three hours on this bug and it turned out the config flag i was trusting didnt even control the thing it said it did. absolutely fuming. anyway fixed now, one line.
Model returned
actually human written
Human written100.00%
Correct. The classifier is reliable on text that resembles what it was trained on.
These are recorded responses, captured on 18 August 2026, not a live call. It got 3 of 6 right. The same behaviour at scale is what Veracia measured.
What it taught me
Verify runtime state, not configured intent
Training loss went to NaN almost immediately. Five hypotheses in a row failed: the precision config, learning-rate warmup, dataloader workers, the attention implementation. The actual cause was that the model loader was silently returning float16 weights regardless of the training flag that was supposed to control precision. I only found it by inspecting the dtype of a loaded tensor directly. The fix was one parameter. The habit it produced is worth more than the fix.
Some bugs only appear statistically
A batch generation script ran through hundreds of successful API calls and then crashed. The provider runs adaptive reasoning on some prompts and not others, so the response occasionally contained a non-text block where the code assumed plain text. Indexing a fixed position works until it does not. Selecting by block type works always.
A high score is a claim, not a conclusion
Shipping at 99.86% and stopping would have been the easy path, and the number would have been technically true the whole time. Building the thing that could disprove it is what turned a metric into an actual understanding of what the model does.
Built with
- PyTorch
- DeBERTa-v3
- scikit-learn
- FastAPI
- Docker
- Caddy
- DigitalOcean