s-chh/PyTorch-Scratch-LLM

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Language: Python - Size: 58.6 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 2 - Forks: 0