waifmark-2

devlog 3:
it’s been even longer but I’ve finally done enough to call this a “devlog”.

basically everything has been fully rewritten in python. A few (quite detrimental) bug fixes are:

• nonexistent error recovery system (i.e. pausing benchmark + invalid responses)
• scoring rubric weights ignored in prompt sent to judge
• save files mislabelling
• A/B scoring only having one slider for better model
• several UI patches (floating navbar -> broken margins)
• added leaderboard from the original index.html currently visible on the github under “Leaderboard”, displaying top scores.

To work on:

• largely inconsistent and seemingly inflated scoring
• check if benchmark can actually be completed, finalise benchmarking criteria + process
• benchmark the rest of the models
  i hope this can be done within 2 more devlogs.

Image 1:
much more simplified run system (compared to before).
Image 2:
the og graph, now in v2!

Dev2

It’s been so long but the UI looks a little better now (?) The process has been basically fully streamlined:

1. HF model download
   -> pick quant/download type
2. Benchmarking
   -> start server -> run benchmark, score/100 evaluation, ETA and live console monitoring
3. Result auditing (Will continue to work on this)
   -> human review responses flagged by LLM judge
4. Results showcase/leaderboard
   -> WIP

Progress related to benchmarking:

• 3x question count compared to Waifmark 1
• ~2x faster parallel automated scoring
• memory system (short/long ctx), agentic toolcalling (basic file reading/shell/research) functions fully implemented as part of local agentic benchmarking.

Devlog 0.1

(For a future project.)
I’m trying out LLM benchmarking on my M1 Macbook Pro that probably needs a break!

• Prior to Stardance -
  Waifmark 1 was a benchmark testing local agentic capabilities and speech persona of small locally hosted (V)LLMs.
  However, my benchmarking process for Waifmark 1 was unstandardised and troublesome, and I kept all the data in excel out of all places.

• Current stage -
  Waifmark 2 is in the works. The benchmark is now evaluated by an LLM-as-a-Judge that can flag and pass low-confidence outputs for human review (as is industry standard).

Using the wonderful streamlit library I built a basic app that can download you a model from hf, serve the model and benchmark it in 3 steps. Unfortunately I cannot show any more behind the process as of now, but I’m very excited to join Stardance and to see what changes can be observed moving from Waifmark 1 -> 2!