TeenyTinyStories
By
@yf
- 7 Devlogs
- 30 Total hours
4h 26m 54s logged
Evals are done
Finally, after two days of tweaking values and rewriting evaluation prompts, the eval is all done. As per the original paper I employed a GPT-Eval adjacent score (albeit with Ministral-14B) as the LLM judge) and an Elo system (with GPT 5.6 Luna as the judge). The results did significantly corroborate with each other, which validates our conclusion that TTS-1M did as well or exceeded (slightly) as TS-1M whilst TTS-33M was poorer in performance compared to TS-33M though still competitive. Both the 33M models significantly outcompeted the 1M models, as expected and serves as an auxiliary validation.
4h 44m 23s logged
4h 0m 1s logged
Models are finished
The models have officially been fully trained and are ready for evaluation. I trained a 1M and a 33M (nominal) parameter model on the TTS dataset; at validation, shows much promise. I also drafted the Methods section and the Related Works section. I will continue to work on the evaluation section and the discussion section, before writing the final paper.
4h 21m 40s logged
On the Models
The 1M model is finished training with tts 1.9870 / 7.29 (4.02M tok), ts_cap 2.3691 / 10.69, ts_full 2.3852 / 10.86. It cost me around 10 USD but I am happy with its results. The 33M model is now undergoing training with ETA in around 10 hours. The models themselves use a GPT-Neo architecture with the EleutherAI/GPT-Neo-125M tokenizer, as per the original paper. I also worked on the paper, namely the methodology section. I don’t know if I can do the 3/9/28M models however due to budget constraints but I will try.
3h 19m 4s logged
3h 6m 58s logged
Devlog 2
In here, I introduce you to the official generation pipeline.I used the TinyStories-Instruct-33M checkpoint as the progenitor model. It accepts a structured header as follows:
Words: verb, noun, adjective
Features: X, Y [optional]
Story:
The words were chosen from the original TS word list from TinyStories-Instruct-valid.txt. The features were derived from an optional feature set, sampled from the empirical distributions in the same TinyStories-Instruct data. Quality control was also enforced by rejecting incomplete stories and stories that were too long or too short. I wrote an .ipynb and began generating stories in batches of 1024 on a rented 4090 (CUDA) and my M1 MacBook Pro (MPS)! Should be ready in around 8 hours.
6h 10m 40s logged
1 On TinyStories
TinyStories (Eldan & Li, 2023) represented a significant breakthrough in the field of language model training, providing evidence that an synthetic (i.e. LLM-generated) corpus could be used to efficiently train a Small Language Model (SLM) fluent in English and basic story generation. The pipeline is direct: Eldan and Li seeded ChatGPT (3.5 and 4) with story generation prompts, generating over 470 M tokens (~2.1 million stories), before training a 1, 3, 9, 28 and 33M SLM. Through a range of evaluation metrics, they demonstrated that these models could output fluent English, and for larger and more layered models, could generate semi-coherent stories.
2 On TeenyTinyStories
Now, I propose a step further: what happens if we take the Instruct variant of the official checkpoint, TinyStories-Instruct-33M instead of a frontier model like the GPT series, to generate a new corpus of data and train the same models on this corpus instead? This is the core of this project.Note: I understand that this is fundamentally a research project rather than produce-a-usable-software type thing you see on Stardance, so I will produce a bunch of usables, including a conversation-tuned instruct variant of the 33M which you can chat with in your browser in the form of a website. The core results, however, will be in the form of an ArXiV preprint before I submit to NHSJS.
What I did
After I skimmed through the 2023 paper, I summarised their corpus generation pipeline as follows:The generator used GPT-3.5 and GPT-4 via the OpenAI APIPrompt composition composed of a list of required words (one verb, one noun, one adjective), and an (optional) story feature (i.e. dialogue, bad ending, moral value, plot twist…) to generate a story. Example: “Write a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand.The story should use the verb “decorate”,the noun “thunder”and the adjective “ancient”.The story should have the following features:the story should contain at least one dialogue,the story has a bad ending.Remember to only use simple words!”Accordingly, I wrote the draft introduction and the Related Works section, and planned the Methodology.
