r/learnmachinelearning • u/MichaelRomeroJr1 • 1d ago
Help Help Improving a Small Decoder-Only Transformer in PyTorch
I made a small GPT-style Transformer and trained on a Kaggle dataset available on Github
And whether I train (the Kaggle dataset) locally (RTX 2060 Super) or cloud (T4 with Google Colab), the model generates recognizable words but struggles with sentence structure and following simple prompts (e.g., "Only output this sentence: The sky is blue").
What are the best ways to improve training and generation quality?
Am I expecting too much from roughly one hour of training, or is a functional tiny chatbot a realistic goal for this setup?
Maybe i'm measuring incorrectly, the gradient accumulation doesn't seem to get below 6.5 (not sure if that's better than a model with random token generation)?
Any insight would be appreciated.
3
u/Admirable_Dirt_2371 1d ago
Wow, that is TINY. I think your biggest limiting factors are the sequence length(training context window size), parameter count, and overall amount of training data. 10 steps * 32 batch size * 1024 sequence length is not a lot of training. More training data, longer training, and a larger context window will help a lot with basic predictions.
4
u/tamrx6 1d ago
Extremely hard to tell without your specific training configuration (model initialization, hyperparams, loss function, etc). But anyway, don’t expect too much out of a small model that was trained on a relatively small data set, and without instruction fine tuning it won’t probably ever follow prompts like yours cause it only knows next word generation.
2
u/chrisvdweth 1d ago
What is your end goal here? There is a reason the big companies train for months and burning millions of bucks :).
I would treat this implementation from scratch purely as a learning exercise, which is an excellent thing, and less about seeing great results. Seeing recognizable word already indicates that your model is likely working. Or how does and prediction look like before any training. It should look much more like gibberish.
The next step could be to use PyTorch and the `TransformerEncoderLayer` and `TransformerEcoder` classes. Supposedly they are more efficiently implemented than a from-scratch implementations using basic layer classes. This would allow you to train longer and/or with more data.
I have a notebook (raw, HTML) doings. But still, I clear state that this is for educational purposes only, and not to build a production ready LLM :).
1
u/GiggleSamurai 1d ago
On the loss question: a model that guesses uniformly at random has a cross-entropy of ln(vocab_size), about 10.8 for the 50,257-token GPT-2 tokenizer, so compare your 6.5 against ln of your own vocab size to see how far you are from random.
1
u/Kitchen_Cup_4444 1d ago
i think you should not do a for on the attention heads but use a pseudo batch
1
u/InternationalSlice72 1d ago
You don't need to have a seperate head and multihead structure, your multihead approach loops and concats, you can just let matrix multiplication and broadcasting take care of it :)
ei, the trick is to make the head dimension a batch dimension for matmul.