LoRA
Definition
Low-Rank Adaptation. A parameter-efficient fine-tuning technique that injects small rank-decomposition matrices into transformer layers.
Example Case
Training a 70MB LoRA adapter for Mistral 7B to write SQL queries.
Low-Rank Adaptation. A parameter-efficient fine-tuning technique that injects small rank-decomposition matrices into transformer layers.