1 On TinyStories

TinyStories (Eldan & Li, 2023) represented a significant breakthrough in the field of language model training, providing evidence that an synthetic (i.e. LLM-generated) corpus could be used to efficiently train a Small Language Model (SLM) fluent in English and basic story generation. The pipeline is direct: Eldan and Li seeded ChatGPT (3.5 and 4) with story generation prompts, generating over 470 M tokens (~2.1 million stories), before training a 1, 3, 9, 28 and 33M SLM. Through a range of evaluation metrics, they demonstrated that these models could output fluent English, and for larger and more layered models, could generate semi-coherent stories.

2 On TeenyTinyStories

Now, I propose a step further: what happens if we take the Instruct variant of the official checkpoint, TinyStories-Instruct-33M instead of a frontier model like the GPT series, to generate a new corpus of data and train the same models on this corpus instead? This is the core of this project.Note: I understand that this is fundamentally a research project rather than produce-a-usable-software type thing you see on Stardance, so I will produce a bunch of usables, including a conversation-tuned instruct variant of the 33M which you can chat with in your browser in the form of a website. The core results, however, will be in the form of an ArXiV preprint before I submit to NHSJS.

What I did

After I skimmed through the 2023 paper, I summarised their corpus generation pipeline as follows:The generator used GPT-3.5 and GPT-4 via the OpenAI APIPrompt composition composed of a list of required words (one verb, one noun, one adjective), and an (optional) story feature (i.e. dialogue, bad ending, moral value, plot twist…) to generate a story. Example: “Write a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand.The story should use the verb “decorate”,the noun “thunder”and the adjective “ancient”.The story should have the following features:the story should contain at least one dialogue,the story has a bad ending.Remember to only use simple words!”Accordingly, I wrote the draft introduction and the Related Works section, and planned the Methodology.