Devlog #005

Hey guys! Final devlog…

Today I made the UI to help people better understand how the network works. Then I wrote the README and pushed everything to GitHub.

The project is done!

Thank you for following this little journey.

Devlog #004

Hey everyone!

Today I made a major upgrade to the neural network. The biggest additions were Dropout and Batch Normalization, which help the model generalize better and make training more stable. I also redesigned the architecture, making it much deeper and larger than before.

Implementing these features wasn’t easy, it took a lot of debugging and experimentation, but the results were definitely worth the effort.

Results

Accuracy: 98.26%

Training is now noticeably slower due to the increased complexity of the network, but I’m very happy with the improvement in performance.

The next step is to build a clean and interactive UI so that anyone can better understand how a neural network learns and makes predictions in real time.

Thanks for reading, and see you in the next devlog!

Devlog #003

Welcome back to my neural network devlog!

In this devlog I focused on improving the training process by implementing mini-batch training and the Adam optimizer.

Instead of updating the network once per epoch using the entire dataset, I now split the training data into batches of 64 images. This allows the model to update its weights much more frequently, making training faster and improving convergence.

I also replaced standard Gradient Descent with Adam. Implementing it from scratch in NumPy required adding the first and second moment estimates for every weight and bias, along with bias correction. Because of Adam, I also reduced the learning rate from 0.1 to 0.001.

The results were a huge improvement: without changing the network architecture (784 → 32 → 10), the test accuracy increased from 80% to 96%, while training became much more stable.

Next, I want to add:

• Larger and deeper network architectures
• Dropout and batch normalization
• A simple UI to visualize the network in real time

Thanks for reading!

Devlog #002

Today I continued working on my neural network project. After implementing a simple XOR model, I wanted to move to something closer to a real machine learning problem, so I started working with the MNIST dataset.

The goal was to make a neural network capable of recognizing handwritten digits from images. I used TensorFlow only to load the dataset, while the entire neural network implementation was written from scratch using only NumPy.

I started by preprocessing the data, converting the 28x28 pixel images into 784 input values and normalizing the pixel values between 0 and 1. I also implemented one-hot encoding for the labels, so the network could classify the 10 possible digits.

For the model, I created a simple Multi-Layer Perceptron (MLP) with:

• an input layer of 784 neurons
• a hidden layer with 32 neurons using the ReLU activation function
• an output layer with 10 neurons using Softmax for classification

I implemented the forward propagation, cross-entropy loss function, backpropagation, and gradient descent completely manually with NumPy.

The most interesting part was seeing how the formulas I studied previously actually translate into code: matrix multiplications for the layers, derivatives for backpropagation, and gradient updates for learning the weights.

The model is still very simple, but it can already reach around 78-80% accuracy on MNIST, which was a great result considering that everything was implemented from scratch.

In the next devlogs, I want to improve the project by adding:

• mini-batch training for better and faster learning
• the Adam optimizer instead of simple gradient descent
• larger and deeper networks
• dropout and batch normalization
• UI to represent how the network works.

Thank you for reading!

Devlog #001

Today I started a pretty big project. I really like math, and I thought this could be the perfect way to combine it with Python.

I started by learning how neural networks work and then moved on to the math behind them. I didn’t just want to memorize the formulas, I wanted to actually understand them, because without understanding the theory, implementing them in code becomes much harder (if not impossible).

The hardest part was, by far, understanding how to implement backpropagation and how all the mathematical concepts connect together.

To finish today’s coding session, I created a small demo to test if I had actually understood everything. I decided to build a simple neural network that predicts the XOR gate (a problem that cannot be solved using a linear function).

For this model, I used Xavier initialization, the hyperbolic tangent (tanh) activation function for the hidden layer, a sigmoid activation function for the output layer, and Binary Cross-Entropy (BCE) as the loss function.

In the next devlog, I will learn how to load and start processing the MNIST dataset.

Thank you for reading!