Just finished pretraining my small language model and it’s yapping some non sense (see images below)

Still it gets the right words combinations so I have a lot of hope for the finetuning.

I went with a regular GPT architecture but reduced the number of transformer blocks and heads to 6 on this model and the embedding dim to 300 so that the training is fast enough (I’m using a NVIDIA A100 with lightning ai’s free credits 🙏) and doesn’t burn too much credits.
Another caveat is that the context is only 256 tokens, I’ll be looking at improving that once I have a correct model.

To give an idea of the size of the model, this one is roughly 36M parameters while GPT-2 smallest model was 124M parameters and GPT-5.5 is around 9.7T parameters.

The next step is gonna be to finetune the model so that it actually answers the question instead of predicting some non sense from the training data.
While I’m working on that I’m also pretraining a bigger model (8 transformer blocks, 8 heads, an embedding dimension of 512 and a bigger batch size) on a H100 (burning my last monthly credits before they expire 🙏🔥).