Search

Word Search

Information System News

Weight-Only Quantization
Rick W

Weight-Only Quantization

Overview This post is divided into five parts; they are: • Asymmetric Affine Quantization • Post-Training Quantization with PyTorch • GPTQ and AWQ • Quantizing a Model to GGUF • Evaluating Weight-Only Quantization Note: The code examples in this post use TorchAO and require a CUDA-enabled PyTorch installation.
Previous Article The Complete Guide to LLM Evaluation
Next Article Serving Scikit-LLM Pipelines as REST APIs with FastAPI
Print
4