Quantization
Definition
Reducing the precision of model weights (e.g. converting float16 to int4) to reduce RAM size and increase inference speed.
Example Case
Running a Llama 3 model quantized to GGUF format on a local Mac with 16GB RAM.