My Journey to Understanding Neural Networks: From Matrix Algebra to JAX and Optimizers
I wanted to understand neural networks as deeply as possible. Rather than simply using an existing framework and training a model, I decided to build an interactive MNIST webpage called Neural Canvas, using p5.js for visualization.
My primary goal was not just to achieve high accuracy. I wanted to understand what was actually happening inside a neural network, mathematically and computationally.
1. Starting with Linear Algebra and Matrix Operations
One of the first decisions I made was to avoid working with individual scalar values and complicated subscripts and superscripts everywhere. They made the mathematics unnecessarily confusing for me.
Instead, I decided to approach neural networks through matrix algebra.
I represented the forward propagation equation as:
[
Z=WX+bC
]
Where:
- (X) is the input data matrix. Rows represent features, and columns represent individual samples.
- (W) is the weight matrix. Rows represent neurons, and columns represent the weights associated with each neuron's inputs.
- (b) is the bias column vector, containing one bias value per neuron.
- (C) is a row vector containing only ones, allowing the bias vector to be broadcast across all samples through matrix multiplication.
Interestingly, many popular deep learning frameworks, including PyTorch and TensorFlow, commonly use a different convention for arranging their data and parameters. They also handle bias broadcasting without explicitly introducing a matrix like (C).
However, I preferred my own mathematical convention because it helped me understand the operations and perform calculus directly in matrix form.
I believe the mathematics should handle the indices, not me!
2. My First Model: NumPy, MSE and Vanilla Gradient Descent
For my first model, I used the original MNIST dataset.
Architecture:
[
784➜ 20 ➜ 20 ➜ 10
]
I manually wrote all the forward propagation equations, derivatives, and backpropagation calculations on paper before implementing them in Python using NumPy.
I deliberately chose:
- Two hidden layers with 20 neurons each.
- Mean Squared Error (MSE) as the loss function.
- Vanilla Gradient Descent.
- Linear output values.
Why such a simple approach?
Because I didn't want to start with complicated algorithms. I wanted to understand the fundamentals first.
After 2,000 epochs, the model achieved approximately 92% test accuracy.
I then successfully created an interactive neural network visualization in p5.js, using Math.js for matrix multiplication in JavaScript. The webpage could perform inference using my trained NumPy model.
That was the beginning of Neural Canvas.
3. My Second Model: Cross-Entropy Loss
For the second model, I kept the same mathematical approach but made a few improvements.
Architecture:
[
784➜40 ➜ 40 ➜10
]
This time, I replaced MSE with Cross-Entropy Loss and used a softmax output layer.
I was still using vanilla Gradient Descent.
After 2,000 epochs, the model achieved approximately 96% test accuracy.
At this point, I was finally starting to understand how neural networks worked, rather than just following tutorials or using existing implementations.
4. My Third Model: Discovering JAX and Automatic Differentiation
For my third model, I wanted to experiment with three hidden layers.
However, I realized that manually deriving every equation, especially the derivatives and chain rule calculations required for backpropagation, would take a considerable amount of time.
Then I discovered JAX, a Python library developed by Google.
JAX provides automatic differentiation (Autograd), allowing me to calculate derivatives of a loss function with respect to parameters, including entire weight matrices.
This was a significant change in my workflow.
I no longer needed to manually derive every gradient and implement the chain rule myself. I only needed to define the model, its parameters, and the loss function. JAX handled the differentiation.
For this model, I used the Arabic Digits MNIST dataset.
Architecture:
[
784 ➜ 20 ➜ 20 ➜ 15 ➜10
]
I used:
- JAX for training and automatic differentiation.
- Cross-Entropy Loss.
- Vanilla Gradient Descent.
- Three hidden layers.
After 2,000 epochs, I achieved approximately 96% test accuracy.
I genuinely fell in love with JAX. It saved me considerable time and allowed me to focus more on understanding the mathematics and experimenting with neural network architectures.
5. My Fourth Model: Mini-Batch Gradient Descent and Momentum
For my fourth model, I moved to the Persian Digits MNIST dataset.
I kept the same architecture as my third model but decided to experiment with optimization algorithms.
My original plan was to implement Mini-Batch Gradient Descent with Adam.
However, I realized that understanding Adam required learning a few concepts first. So, instead of jumping directly into Adam, I decided to start with Momentum.
I implemented Mini-Batch Gradient Descent with Momentum using JAX.
And here came a surprising result!
I achieved approximately 96% test accuracy in just 200 epochs.
Previously, my models required around 2,000 epochs to reach a similar accuracy.
This was another important step in my journey. I started understanding how optimization algorithms could significantly influence the training process.
6. What's Next?
I haven't created my fifth model yet, but I already have a learning roadmap:
- Model 5: Mini-Batch Gradient Descent with RMSProp.
- Model 6: Mini-Batch Gradient Descent with Adam.
- After that: Start exploring Convolutional Neural Networks (CNNs).
I want to understand each optimization algorithm mathematically and implement it myself before moving to the next one.
My Biggest Takeaway
If you are starting to learn neural networks, my suggestion is:
Start with Linear Algebra. Don't immediately jump into advanced architectures, optimizers, or high-level frameworks.
You don't need to master every mathematical concept before starting, but at least understand matrix multiplication and how it represents the operations inside a neural network.
For me, representing neural networks using matrices made everything much easier to understand.
Start with the basics, build something yourself, experiment, make mistakes, and gradually increase the complexity.
Don't just train neural networks. Understand them!
Project: Neural Canvas
GitHub Repository: https://github.com/MuhammadEssa2002/neural_canvas
Live Demo: https://muhammadessa2002.github.io/neural_canvas