Devlog #003

Welcome back to my neural network devlog!

In this devlog I focused on improving the training process by implementing mini-batch training and the Adam optimizer.

Instead of updating the network once per epoch using the entire dataset, I now split the training data into batches of 64 images. This allows the model to update its weights much more frequently, making training faster and improving convergence.

I also replaced standard Gradient Descent with Adam. Implementing it from scratch in NumPy required adding the first and second moment estimates for every weight and bias, along with bias correction. Because of Adam, I also reduced the learning rate from 0.1 to 0.001.

The results were a huge improvement: without changing the network architecture (784 → 32 → 10), the test accuracy increased from 80% to 96%, while training became much more stable.

Next, I want to add:

• Larger and deeper network architectures
• Dropout and batch normalization
• A simple UI to visualize the network in real time

Thanks for reading!