About the Project

An honest study rather than a product claim. It asks whether a cheap CPU model can replace a 0.6B neural monitor for early failure detection, and answers that it recovers about half the estimated token savings at 2.1 KB and 0.18 ms on one CPU core. The predictive signal is real and traces to declining action novelty rather than repeated errors, but the pre-registered operating point a shipping product would need was not reached, and the writeup reports that as found.

Key Features

• 33 prefix-only trajectory features over the first N turns, built from 237,132 turn records and guarded by a leakage suite that shuffles, truncates and extends the unseen tail
• Repository-grouped cross-validation with confidence intervals resampling repositories rather than rows
• Calibrated logistic model at 2.1 KB scoring one session in 0.18 ms p50 on a single CPU core
• AUROC 0.677 [0.651, 0.697] against three trivial baselines at 0.500-0.521
• Estimated 8.4% token savings against the 14.6-20.4% reported for a 0.6B neural monitor; the corpus records no token counts, so savings are derived from content length
• Success criteria pre-registered in source control before training; the 0.25-recall gate was missed at 0.191
• Every output separates measured counts from model estimates, and labels which is which
• Measured feature drift between training and live transcripts; the six largest shifts exceed 1.7 standard deviations and two of 33 features fall outside the training range, so cross-scaffold accuracy is reported as unmeasured rather than assumed
• Local CLI and MCP server, no LLM anywhere in the inference path