Large Language Models (LLMs) rely on weight matrices to calculate probabilities and generate next token. Traditionally, fine-tuning these models to specific tasks requires "full fine-tuning" which is a computationally expensive process that demands massive hardware memory to update millions or billions of individual weights.

This presentation demystifies Low-Rank Adaptation (LoRA), a leading Parameter-Efficient Fine-Tuning (PEFT) technique that completely bypasses this hardware bottleneck. We will explore the mathematical "magic" that allows to achieve the exact same probability redistributions as full fine-tuning, while updating only a tiny fraction of the model's weights.

Attendees will be guided through the complete lifecycle of this process. We will map out how data moves through forward passes, loss calculations, and backward passes across multiple training epochs, before finally exploring the inference stage where these newly customized models are efficiently loaded and served. At the end, there is a demonstration of finetuning an LLM while showing the difference in generated output.