Premature optimization is the root of all evils

I still fell in the trap…

I struggled for weeks to produce a model that can generate coherent text. Now all what is left is to ship or perhaps even fine-tune and be done. But nooooo, we need to atleast make PeRfOrmAnt.

Ended adding some mods to the original GPT-2. I found that there’s even a whole community around this .
I gained a lot from them.

Now, the project is a Frankenstein DeepSeek.

Time will tell if this was a good idea (-_-)

Here’s a picture of the last training round. Things went quite well despite a bug affecting the learning rate.