Evals are done

Finally, after two days of tweaking values and rewriting evaluation prompts, the eval is all done. As per the original paper I employed a GPT-Eval adjacent score (albeit with Ministral-14B) as the LLM judge) and an Elo system (with GPT 5.6 Luna as the judge). The results did significantly corroborate with each other, which validates our conclusion that TTS-1M did as well or exceeded (slightly) as TS-1M whilst TTS-33M was poorer in performance compared to TS-33M though still competitive. Both the 33M models significantly outcompeted the 1M models, as expected and serves as an auxiliary validation.

While we’re waiting…

Models are finished

The models have officially been fully trained and are ready for evaluation. I trained a 1M and a 33M (nominal) parameter model on the TTS dataset; at validation, shows much promise. I also drafted the Methods section and the Related Works section. I will continue to work on the evaluation section and the discussion section, before writing the final paper.

On the Models

The 1M model is finished training with tts 1.9870 / 7.29 (4.02M tok), ts_cap 2.3691 / 10.69, ts_full 2.3852 / 10.86. It cost me around 10 USD but I am happy with its results. The 33M model is now undergoing training with ETA in around 10 hours. The models themselves use a GPT-Neo architecture with the EleutherAI/GPT-Neo-125M tokenizer, as per the original paper. I also worked on the paper, namely the methodology section. I don’t know if I can do the 3/9/28M models however due to budget constraints but I will try.

Docs r out. Just b4 release, I will compile a windows compatible exe too, since I am unsigned on mac!

Devlog 3

Full corpus generation finished ASAP on 4090. You can find the TTS corpus on ableian/teenytinystories on Huggingface. Now, I followed the architecture specified by Eldan and Li and started training the 1M/3M and 9M models. Will finish in 16 hours, so I’m excited!

Devlog 2

In here, I introduce you to the official generation pipeline.I used the TinyStories-Instruct-33M checkpoint as the progenitor model. It accepts a structured header as follows:

Words: verb, noun, adjective
Features: X, Y [optional]
Story:

The words were chosen from the original TS word list from TinyStories-Instruct-valid.txt. The features were derived from an optional feature set, sampled from the empirical distributions in the same TinyStories-Instruct data. Quality control was also enforced by rejecting incomplete stories and stories that were too long or too short. I wrote an .ipynb and began generating stories in batches of 1024 on a rented 4090 (CUDA) and my M1 MacBook Pro (MPS)! Should be ready in around 8 hours.

1 On TinyStories

TinyStories (Eldan & Li, 2023) represented a significant breakthrough in the field of language model training, providing evidence that an synthetic (i.e. LLM-generated) corpus could be used to efficiently train a Small Language Model (SLM) fluent in English and basic story generation. The pipeline is direct: Eldan and Li seeded ChatGPT (3.5 and 4) with story generation prompts, generating over 470 M tokens (~2.1 million stories), before training a 1, 3, 9, 28 and 33M SLM. Through a range of evaluation metrics, they demonstrated that these models could output fluent English, and for larger and more layered models, could generate semi-coherent stories.

2 On TeenyTinyStories

Now, I propose a step further: what happens if we take the Instruct variant of the official checkpoint, TinyStories-Instruct-33M instead of a frontier model like the GPT series, to generate a new corpus of data and train the same models on this corpus instead? This is the core of this project.Note: I understand that this is fundamentally a research project rather than produce-a-usable-software type thing you see on Stardance, so I will produce a bunch of usables, including a conversation-tuned instruct variant of the 33M which you can chat with in your browser in the form of a website. The core results, however, will be in the form of an ArXiV preprint before I submit to NHSJS.

What I did

After I skimmed through the 2023 paper, I summarised their corpus generation pipeline as follows:The generator used GPT-3.5 and GPT-4 via the OpenAI APIPrompt composition composed of a list of required words (one verb, one noun, one adjective), and an (optional) story feature (i.e. dialogue, bad ending, moral value, plot twist…) to generate a story. Example: “Write a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand.The story should use the verb “decorate”,the noun “thunder”and the adjective “ancient”.The story should have the following features:the story should contain at least one dialogue,the story has a bad ending.Remember to only use simple words!”Accordingly, I wrote the draft introduction and the Related Works section, and planned the Methodology.

The dmg is ready! now, website for product before final shipping…

Prepping for final checks before shipping! Made some assets as well.

26/26 pass! Mutex connections and database sanitisation is certified and up and running. SO far on the release checklist, half of everything is done!

Did comprehensive backend stress testing especially on the health+integrity of backend database structures by forcefully inserting malformed data. Identified around 2-3 vulnerabilities and 2 blunt failures and I will resolve that in the backend by force-sanitising all data that is to be entered

Made the search bar in toolkits functional and debugged the css-bug in which the model-detail was not aligned with the styles of model-card classes among other bug hunting work

Major housekeeping, css bug fixing too

Added comprehensive error coding (documentation website tba)

Stopping logic refined, huge unification of toolcall/send direct logic and further housekeeping incl/ with deletion of target/ recompilation, and headless testing (heterogeneous AI testing) and major user testing

Now toolkits are enable-able (they are too premature for production, they are a beta-feature for the alpha release, I couldn’t really be bothered with more hours of edge case testing and fixing of this specific singular function) and I also added a search bar.

Housekeeping is taking way too long… did some style unifying, etc. Did minor structural edits for edge casing, next devlog I will delete my cache and re-do it to test the full experience! Also, collapsable sidebar!

Did a lot of housekeeping and stylistic edits! Preparing for alpha release! (Accidentally coded for too long before a devlog, oopsies.)

yay first in australia (55h this week). Anyway did a bunch of stuff including housekeeping and error catching, introducing error codes E04 and E05; more info will come as we get closer to github release and alpha release.