Gptq

orchestra-research/ai-research-skills/10-optimization/gptq

作者 orchestra-research773a52944ba4MIT13K 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.

仅含说明Software DevelopmentAI & Agents
AI 生成的概览

指导对大型语言模型进行 4 位 GPTQ 训练后量化，以降低显存占用并加速推理。

功能
说明如何安装并使用 AutoGPTQ，通过分组量化把大型语言模型压缩到 4 位，加载已量化模型并执行推理。内容涵盖量化配置、ExLlamaV2、Marlin 与 Triton 等内核后端、transformers 与 PEFT/QLoRA 集成、多 GPU 与 CPU 卸载，以及显存、速度和困惑度的基准表格。它产出的是量化模型文件与部署指导，本身不执行任何操作。
适用场景
适用于在显存有限或消费级 GPU 上运行 70B、405B 等大模型，希望在精度损失很小的前提下将显存减少约 4 倍，或需要比 FP16 更快的推理速度时。也适合在 GPTQ、AWQ 与 bitsandbytes 之间做选择，或用 QLoRA 微调量化模型。
运行要求
需要 Python 环境及 auto-gptq、transformers、optimum、peft 等依赖，并需要支持 CUDA 的 NVIDIA GPU 进行量化与推理；Triton 仅限 Linux，Marlin 需要 Ampere 或更新架构。需要网络访问以从 HuggingFace 下载模型和校准数据集。不附带脚本，只有参考文档。

GPTQ (Generative Pre-trained Transformer Quantization)

Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.

When to use GPTQ

Use GPTQ when:

  • Need to fit large models (70B+) on limited GPU memory
  • Want 4× memory reduction with <2% accuracy loss
  • Deploying on consumer GPUs (RTX 4090, 3090)
  • Need faster inference (3-4× speedup vs FP16)

Use AWQ instead when:

  • Need slightly better accuracy (<1% loss)
  • Have newer GPUs (Ampere, Ada)
  • Want Marlin kernel support (2× faster on some GPUs)

Use bitsandbytes instead when:

  • Need simple integration with transformers
  • Want 8-bit quantization (less compression, better quality)
  • Don't need pre-quantized model files

Quick start

Installation

bash
# Install AutoGPTQpip install auto-gptq
# With Triton (Linux only, faster)pip install auto-gptq[triton]
# With CUDA extensions (faster)pip install auto-gptq --no-build-isolation
# Full installationpip install auto-gptq transformers accelerate

Load pre-quantized model

python
from transformers import AutoTokenizerfrom auto_gptq import AutoGPTQForCausalLM
# Load quantized model from HuggingFacemodel_name = "TheBloke/Llama-2-7B-Chat-GPTQ"
model = AutoGPTQForCausalLM.from_quantized(    model_name,    device="cuda:0",    use_triton=False  # Set True on Linux for speed)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Generateprompt = "Explain quantum computing"inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0]))

Quantize your own model

python
from transformers import AutoTokenizerfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfigfrom datasets import load_dataset
# Load modelmodel_name = "meta-llama/Llama-2-7b-chat-hf"tokenizer = AutoTokenizer.from_pretrained(model_name)
# Quantization configquantize_config = BaseQuantizeConfig(    bits=4,              # 4-bit quantization    group_size=128,      # Group size (recommended: 128)    desc_act=False,      # Activation order (False for CUDA kernel)    damp_percent=0.01    # Dampening factor)
# Load model for quantizationmodel = AutoGPTQForCausalLM.from_pretrained(    model_name,    quantize_config=quantize_config)
# Prepare calibration datadataset = load_dataset("c4", split="train", streaming=True)calibration_data = [    tokenizer(example["text"])["input_ids"][:512]    for example in dataset.take(128)]
# Quantizemodel.quantize(calibration_data)
# Save quantized modelmodel.save_quantized("llama-2-7b-gptq")tokenizer.save_pretrained("llama-2-7b-gptq")
# Push to HuggingFacemodel.push_to_hub("username/llama-2-7b-gptq")

Group-wise quantization

How GPTQ works:

  1. Group weights: Divide each weight matrix into groups (typically 128 elements)
  2. Quantize per-group: Each group has its own scale/zero-point
  3. Minimize error: Uses Hessian information to minimize quantization error
  4. Result: 4-bit weights with near-FP16 accuracy

Group size trade-off:

Group SizeModel SizeAccuracySpeedRecommendation
-1 (per-column)SmallestBestSlowestResearch only
32SmallerBetterSlowerHigh accuracy needed
128MediumGoodFastRecommended default
256LargerLowerFasterSpeed critical
1024LargestLowestFastestNot recommended

Example:

Weight matrix: [1024, 4096] = 4.2M elements
Group size = 128:- Groups: 4.2M / 128 = 32,768 groups- Each group: own 4-bit scale + zero-point- Result: Better granularity → better accuracy

Quantization configurations

Standard 4-bit (recommended)

python
from auto_gptq import BaseQuantizeConfig
config = BaseQuantizeConfig(    bits=4,              # 4-bit quantization    group_size=128,      # Standard group size    desc_act=False,      # Faster CUDA kernel    damp_percent=0.01    # Dampening factor)

Performance:

  • Memory: 4× reduction (70B model: 140GB → 35GB)
  • Accuracy: ~1.5% perplexity increase
  • Speed: 3-4× faster than FP16

High accuracy (3-bit with larger groups)

python
config = BaseQuantizeConfig(    bits=3,              # 3-bit (more compression)    group_size=128,      # Keep standard group size    desc_act=True,       # Better accuracy (slower)    damp_percent=0.01)

Trade-off:

  • Memory: 5× reduction
  • Accuracy: ~3% perplexity increase
  • Speed: 5× faster (but less accurate)

Maximum accuracy (4-bit with small groups)

python
config = BaseQuantizeConfig(    bits=4,    group_size=32,       # Smaller groups (better accuracy)    desc_act=True,       # Activation reordering    damp_percent=0.005   # Lower dampening)

Trade-off:

  • Memory: 3.5× reduction (slightly larger)
  • Accuracy: ~0.8% perplexity increase (best)
  • Speed: 2-3× faster (kernel overhead)

Kernel backends

ExLlamaV2 (default, fastest)

python
model = AutoGPTQForCausalLM.from_quantized(    model_name,    device="cuda:0",    use_exllama=True,      # Use ExLlamaV2    exllama_config={"version": 2})

Performance: 1.5-2× faster than Triton

Marlin (Ampere+ GPUs)

python
# Quantize with Marlin formatconfig = BaseQuantizeConfig(    bits=4,    group_size=128,    desc_act=False  # Required for Marlin)
model.quantize(calibration_data, use_marlin=True)
# Load with Marlinmodel = AutoGPTQForCausalLM.from_quantized(    model_name,    device="cuda:0",    use_marlin=True  # 2× faster on A100/H100)

Requirements:

  • NVIDIA Ampere or newer (A100, H100, RTX 40xx)
  • Compute capability ≥ 8.0

Triton (Linux only)

python
model = AutoGPTQForCausalLM.from_quantized(    model_name,    device="cuda:0",    use_triton=True  # Linux only)

Performance: 1.2-1.5× faster than CUDA backend

Integration with transformers

Direct transformers usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load quantized model (transformers auto-detects GPTQ)model = AutoModelForCausalLM.from_pretrained(    "TheBloke/Llama-2-13B-Chat-GPTQ",    device_map="auto",    trust_remote_code=False)
tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-Chat-GPTQ")
# Use like any transformers modelinputs = tokenizer("Hello", return_tensors="pt").to("cuda")outputs = model.generate(**inputs, max_new_tokens=100)

QLoRA fine-tuning (GPTQ + LoRA)

python
from transformers import AutoModelForCausalLMfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model
# Load GPTQ modelmodel = AutoModelForCausalLM.from_pretrained(    "TheBloke/Llama-2-7B-GPTQ",    device_map="auto")
# Prepare for LoRA trainingmodel = prepare_model_for_kbit_training(model)
# LoRA configlora_config = LoraConfig(    r=16,    lora_alpha=32,    target_modules=["q_proj", "v_proj"],    lora_dropout=0.05,    bias="none",    task_type="CAUSAL_LM")
# Add LoRA adaptersmodel = get_peft_model(model, lora_config)
# Fine-tune (memory efficient!)# 70B model trainable on single A100 80GB

Performance benchmarks

Memory reduction

ModelFP16GPTQ 4-bitReduction
Llama 2-7B14 GB3.5 GB4×
Llama 2-13B26 GB6.5 GB4×
Llama 2-70B140 GB35 GB4×
Llama 3-405B810 GB203 GB4×

Enables:

  • 70B on single A100 80GB (vs 2× A100 needed for FP16)
  • 405B on 3× A100 80GB (vs 11× A100 needed for FP16)
  • 13B on RTX 4090 24GB (vs OOM with FP16)

Inference speed (Llama 2-7B, A100)

PrecisionTokens/secvs FP16
FP1625 tok/s1×
GPTQ 4-bit (CUDA)85 tok/s3.4×
GPTQ 4-bit (ExLlama)105 tok/s4.2×
GPTQ 4-bit (Marlin)120 tok/s4.8×

Accuracy (perplexity on WikiText-2)

ModelFP16GPTQ 4-bit (g=128)Degradation
Llama 2-7B5.475.55+1.5%
Llama 2-13B4.884.95+1.4%
Llama 2-70B3.323.38+1.8%

Excellent quality preservation - less than 2% degradation!

Common patterns

Multi-GPU deployment

python
# Automatic device mappingmodel = AutoGPTQForCausalLM.from_quantized(    "TheBloke/Llama-2-70B-GPTQ",    device_map="auto",  # Automatically split across GPUs    max_memory={0: "40GB", 1: "40GB"}  # Limit per GPU)
# Manual device mappingdevice_map = {    "model.embed_tokens": 0,    "model.layers.0-39": 0,  # First 40 layers on GPU 0    "model.layers.40-79": 1,  # Last 40 layers on GPU 1    "model.norm": 1,    "lm_head": 1}
model = AutoGPTQForCausalLM.from_quantized(    model_name,    device_map=device_map)

CPU offloading

python
# Offload some layers to CPU (for very large models)model = AutoGPTQForCausalLM.from_quantized(    "TheBloke/Llama-2-405B-GPTQ",    device_map="auto",    max_memory={        0: "80GB",  # GPU 0        1: "80GB",  # GPU 1        2: "80GB",  # GPU 2        "cpu": "200GB"  # Offload overflow to CPU    })

Batch inference

python
# Process multiple prompts efficientlyprompts = [    "Explain AI",    "Explain ML",    "Explain DL"]
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")
outputs = model.generate(    **inputs,    max_new_tokens=100,    pad_token_id=tokenizer.eos_token_id)
for i, output in enumerate(outputs):    print(f"Prompt {i}: {tokenizer.decode(output)}")

Finding pre-quantized models

TheBloke on HuggingFace:

Search:

bash
# Find GPTQ models on HuggingFacehttps://huggingface.co/models?library=gptq

Download:

python
from auto_gptq import AutoGPTQForCausalLM
# Automatically downloads from HuggingFacemodel = AutoGPTQForCausalLM.from_quantized(    "TheBloke/Llama-2-70B-Chat-GPTQ",    device="cuda:0")

Supported models

  • LLaMA family: Llama 2, Llama 3, Code Llama
  • Mistral: Mistral 7B, Mixtral 8x7B, 8x22B
  • Qwen: Qwen, Qwen2, QwQ
  • DeepSeek: V2, V3
  • Phi: Phi-2, Phi-3
  • Yi, Falcon, BLOOM, OPT
  • 100+ models on HuggingFace

References

  • Calibration Guide [blocked] - Dataset selection, quantization process, quality optimization
  • Integration Guide [blocked] - Transformers, PEFT, vLLM, TensorRT-LLM
  • Troubleshooting [blocked] - Common issues, performance optimization

Resources

来源与署名

来源：orchestra-research/ai-research-skills位于10-optimization/gptq提交773a529

许可证： MIT

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 orchestra-research/ai-research-skills 的技能

Speculative Decoding

orchestra-research

指导使用推测解码、Medusa 多头预测和前瞻解码来加速大语言模型推理。

AI & Agents13K3个月前更新

Systems Paper Writing

orchestra-research

为 OSDI、SOSP、ASPLOS、NSDI 等系统会议论文写作提供段落级结构与篇幅分配指导。

Writing & Content13K3个月前更新

Moe Training

orchestra-research

指导使用 DeepSpeed 或 HuggingFace 训练与推理混合专家模型，涵盖路由、负载均衡与专家并行。

AI & Agents13K3个月前更新

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

待分类13K3个月前更新

Model Pruning

orchestra-research

指导使用 Wanda、SparseGPT、幅度剪枝和 N:M 方法压缩 LLM，减小模型体积并加速推理。

AI & Agents13K3个月前更新

Outlines

orchestra-research

指导使用 Outlines 库，通过本地模型进行受结构约束的文本生成。

AI & Agents13K3个月前更新

更多Software Development技能

Playground

anthropics

精选

构建自包含的交互式 HTML 演练场，包含控件、实时预览和可复制的提示词输出。

Software Development37K今天更新

M5 Onboard

anthropics

精选

通过 USB 检测 M5Stack ESP32 开发板，刷写 UIFlow 2.0 固件并安装 MicroPython 应用包。

Software Development37K今天更新

Cardputer Buddy

anthropics

精选

指导迭代 Cardputer-Adv 的 MicroPython 应用包：添加应用、通过 USB 串口推送文件、查看日志并运行 REPL 命令。

Software Development37K今天更新

Web Design Guidelines

vercel-labs

精选

依据从远程来源获取的 Web Interface Guidelines 审查 UI 代码，并以 file:line 形式报告问题。

Software Development32K5周前更新

Vercel React Native Skills

vercel-labs

精选

React Native 与 Expo 最佳实践规则，涵盖列表性能、动画、导航、UI 模式、状态与 monorepo 配置。

Software Development32K5周前更新

Vercel React Best Practices

vercel-labs

精选

Vercel 的 React 与 Next.js 性能指南，以 70 条规则的形式用于编写、审查和重构代码。

Software Development32K5周前更新