On the Models

The 1M model is finished training with tts 1.9870 / 7.29 (4.02M tok), ts_cap 2.3691 / 10.69, ts_full 2.3852 / 10.86. It cost me around 10 USD but I am happy with its results. The 33M model is now undergoing training with ETA in around 10 hours. The models themselves use a GPT-Neo architecture with the EleutherAI/GPT-Neo-125M tokenizer, as per the original paper. I also worked on the paper, namely the methodology section. I don’t know if I can do the 3/9/28M models however due to budget constraints but I will try.