devlog: the 100-hour milestone, a humaneval script blowout, and parsing raw logs

context recap (the architecture so far)

for anyone catching up on the architecture overhaul: my initial ship for thalamus got rejected because the UI looked like an AI wrapper template. instead of shifting pixels, i decided to completely redo the backend engine. i’m training a custom, specialized 32B model optimized explicitly for programming logic—named thalamus-thor-1. general fluff is scrubbed to save weight; a gemini researcher agent sub-team handles web scraping to cover info blind spots.

resource-wise, it’s a tight squeeze. puter.js failed on orchestration, so i’m entirely dependent on a drying runway of AWS credits while waiting on google cloud and cheap amd gpu server approvals. with school starting in two days, i’ve also been preparing a lightweight remote framework to monitor logs and trigger test suites from school lab computers during short periods. earlier tonight, i managed to bypass kaggle’s aggressive session killers, successfully compiled the FP8/4-bit compressed weights on their tpu clusters, and immediately threw the model straight into openai’s standard 164-problem HumanEval benchmark.

here is the live update from late tonight.

---

milestone: 100 hours in the dirt

first off, a quick personal timestamp: as of setting up these evaluation scripts, i have officially crossed the 100-hour mark of active development on this project. hitting triple digits while drowning in memory leaks, credit run-outs, and hardware timeouts is a brutal milestone, but the core engine is standing.

the humaneval script blowout

i watched the model successfully plow past problem 90, and it kept charging all the way through the end of the execution run. but right at the finish line, instead of outputting the final pass@1 score summary, the script threw a fatal calculation error and crashed out.

the evaluation script generated the completions, ran the test assertions, and executed the functional verification, but the section of the code responsible for processing the final combination formula or compiling the macro-averaged percentage broke completely. standard human-eval metric scripts use an unbiased estimator calculation to evaluate pass@k:

$$\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

if the array containing the total number of generated completions ($n$) or the correct passes ($c$) contains a single malformed row, a zero-division error, or an unhandled null exception from an execution timeout, the entire summary wrapper explodes. i’m staring at a broken metrics calculator, but the actual data isn’t lost.

shifting to manual validation

i’m not letting a broken parsing script mask the actual performance metrics of the weights after 100 hours of grind. the script managed to compile and save the fine-grained data out to the raw <input_path>_results.jsonl log file before the summary function choked. each row in that file contains the literal execution results marked cleanly as "passed", "failed", or "timed out".

instead of trying to debug the evaluator framework’s entry points tonight, i’m going to pull the raw JSON lines into a basic python script i’m writing right now. i’ll manually iterate through the rows, count the hard instances of "passed" across the tasks, and divide it by 164 to find the true, absolute pass@1 score myself.

the raw logs are sitting in the directory. writing the cleanup parser now to see what thalamus-thor-1 actually scored.