AI · Technology · 2026-09-09 07:45:00
The author rebuilt an AI code comment classifier on public data after the previous version used private data. The new classifier has 77% balanced accuracy and outputs calibrated probabilities. The confusion matrix shows 73% correct on known-human and 80% on known-robot inputs, with false-positive risk dropping to 5% at high confidence. The dataset was built from permissively licensed repos, taking 2021 commits, stripping comments and having LLMs generate new ones. The author made repeated data mistakes, costing far more than the theoretical $30. Features evaluated include bigwords, wordlen, freqrank, wordfreq, word2gram, charfreq and POS n-grams. The classifier is trained seven ways but Kimi K2.7 and glm overlap other models, so their probability mass is smeared back. The model is 355 kB after quantization. The author won't continue due to sourcehut's new terms.
Auto-generated summary for reference only. RSS