Dear shipwrights

I don’t know if you’ll read this but I still feel like writing it since I’ve nothing else to write.

In response to the feedback provided, here’s what changed:

• I deleted some part of my README as well as corrected grammar mistakes since it seemed to “contain” part written by AI. Honestly the grammar mistakes should’ve given me an alibi but … nevermind.

• I have patched garbage characters generation during inference. I truly hope that was the “bug” you mentioned in your review. If it was not, I’m afraid there’s not more I can do without spending ton of time/money on computing to improve my Language Model output (NOT chatbot). Which would be sad. Really sad. For the planet and for me.

• Detailed AI usage in the project description.

I really hope the third time will pay home 🥲

Dear shipwrights

My Cloudflared tunnel went down and I don’t know why. Thus the link in my shipping devlog is now invalid.

I looked up if there was any way to get a more lasting url and all solutions I found were either paid services (domain name) or required a credit card.

I have neither.

I will wait for your message and try to improve networking skills for the next ship.

Thanks for reading me.

Sadly,
@Definitly-Not-Me

Wait ! shipping require you to post a devlog ?

Bad luck on top of bad luck. I found no cloud hosting platform that check all my requirements so for the time being, I’m a cloud provider :)

For those who will review my project, I ask for your understanding.

Nothing to show for this one. So here is picture of my Language Model mascot.

Patrick Star

Last devlog before shipping:

Against my best wishes, none of my models reached a usable state. Even though their best loss is quite low (2.94 and 2.81) I didn’t manage to reach the finetuning stage.

To continue training I would have to start pouring money to get more compute power. And I don’t have that….

So that’ll be all.

I will finish the UI and deploy locally first until I manage to find a vps before my project get a reviewer.

Premature optimization is the root of all evils

I still fell in the trap…

I struggled for weeks to produce a model that can generate coherent text. Now all what is left is to ship or perhaps even fine-tune and be done. But nooooo, we need to atleast make PeRfOrmAnt.

Ended adding some mods to the original GPT-2. I found that there’s even a whole community around this .
I gained a lot from them.

Now, the project is a Frankenstein DeepSeek.

Time will tell if this was a good idea (-_-)

Here’s a picture of the last training round. Things went quite well despite a bug affecting the learning rate.

Updates and Bloopers

Even though Kaggle runs on Google Collab, it turn out Collab is much more easier to use than Kaggle. To my grand deception, Collab is not as generous as Kaggle. So I will not change my workflow.

I think I also need to change dataset.
Also what’s up with Stardance and all those auxiliary events/challenges ?

Now we’re talking folks

Reached a milestone. After nearly 10 hours of training on OpenwebText, the model can now formulate somewhat grammatically correct sentences.

It’s not over though.

Training was interrupted because apparently kaggle has a time limit for how long a notebook can run (on top on the weekly time limit). Sadly I have no interest in buying cloud compute for that, so I will just look for a richer dataset, reduce the number of epoch and crank up the learning rate.

I’m tired boss

Sorry, stardance but for my own good and sanity, I think I will have to give up or half bake this project. I really tried 😞

Two steps before the throne

Finally deployed the training script to Kaggle. Now all there’s to be is wait and hope.

A quick list of what’s new:

• Divided the training loop in many functions to handle dataloading, parsing, logging and more;
• Managed to find a copy of gutenberg books to serve as dataset;
• Implemented checkpoints in the training so that I will not lose all progress if I run out of compute;
• Dataloading is now done using np.memmap ensuring that only a portion of tokens are loaded in memory;
• Scaled the value of hyprparameters based on configuration of gpt-2 small (my main source is Chatgpt/GLM and a rough guess of number of epochs);
• Fixed a “storage leak” where the best model checkpoints where saved per epoch instead of overwriting previous one (totally my fault)

Next:

• Fix my nonexistant git history while waiting for the training to finish

Small update on building a Language Model

So after getting the architecture down and adding collecting some data, I started the training phase. That was one week ago ….
Training is surprisingly (or not ?) actually more than just shoving in more data and expect it to work. Right now I’m struggling with finding sensible hyperparameters and getting the model to output something coherent.

What’s new

I mainly worked on the training loop and minor optimizations to hidden layers to distract myself while Pytorch was frying my APU. Of course there were also bugs in the code that had to fix on the way.

For monitoring the training I decided to use plain TUI built with Rich but I’m currently adding a csv logging for better history tracking.

At first I was planning to run the training on my laptop but I understood why billions dollar companies needed so many datacenters. Anyway, I deployed it on Kaggle which also was harder than it should be.

Next

• Take a break,
• Improve training loop,
• Deploy on Kaggle
• Forget about it

Devlog 0: Build a Large/Small/Whatever LM

Overview

Hello Stardance! Hello world appetIntern disdainwartsnine.
Those were the first words of my Language Model, am feeling so fogging proud right now. I’m trying to get into ML for the next year (not just because of the hype). For my first projects, I tried building a simple perceptron as well as understanding the algorithm behind Tic-Tac-Toe. Nothing grandiose. Why aim so high for my next project since I’m not that knowledgeable ? Because I’m short of time and there are full tutorial available, that’s all.

Goal & MVP

The MVP goal is to be able to generate coherent text (will probably be gibberish since I’m not a billion dollars corpo) and implement simple RAG to compensate for the lack of intelligence.

I plan to upgrade it as I learn more about LLMs and Transformers.

The Current State

I just got the basic unoptimized architecture down.
It tooks so much time to understand the Transformer architecture and all the maths that goes behind it (thanks to 3Blue1Brown it was manageable). Now I need to train & evaluate the “Advanced Autocomplete Machine” so that It can start acting more like GPT2.

Picture:

Liking my rice ? 😏