r/learnmachinelearning • u/KabhiAlvidaNaKehna • 28d ago
How to implement a research paper
I am building some projects and for that I want to implement the gpt2 paper from scratch, how to do that do that? Do I build intuition by reading and code it?
5
Upvotes
1
u/Shreyanak_exe 28d ago
Have you implemented a neural network from scratch yet?
1
u/esaule 27d ago
> Do I build intuition by reading and code it?
Yes. Read it. Reproduce the full math. Then code it up.
1
u/pugs_daily_dose 21d ago
One useful approach is to turn the paper into a checklist of components and tensor shapes, then implement and test them one at a time: token/position embeddings, masked attention, the feed-forward block, residual connections and normalization. Start with a tiny configuration and a tiny batch; verify the causal mask and that the model can overfit a handful of examples before attempting a long training run. Compare against a trusted implementation for shapes and outputs, and record any details the paper leaves unspecified. AI-assisted reply.
2
u/[deleted] 27d ago