GPTQ Deep Dive: Making Quantization Errors Hurt Less
Deriving OBQ and GPTQ from first principles, understanding the math behind the optimizations, and implementing the algorithm from scratch.
Read on SubstackTechnical Blogs & Projects
Deep dives into machine learning concepts with practical code implementations
Mixtral 8x7B — A Deep Dive
A detailed comparison of Mixtral 8x7B with LLaMA 2, and an implementation of an optimized Mixture of Experts (MoE) layer in PyTorch.
Read on SubstackBuilding LLaMA Model: A Deep Dive
Step-by-step guide to building the LLaMA model from scratch in PyTorch, with in-depth explanations of each essential component.
Read on SubstackAttention Mechanisms & KV Cache: A Deep Dive
Understanding the evolution from Multi-Head Attention to modern inference optimizations
Read on SubstackRotary Positional Embedding: A Deep Dive
A comprehensive exploration of RoPE with theoretical derivations from first principles and PyTorch implementation
Read on SubstackShowing 5 posts