AWQ vs GPTQ vs FP8: Which Quantization for Your GPU

AWQ and GPTQ produce W4A16 checkpoints: 4-bit weights, 16-bit activations, group size usually 128. FP8 W8A8 gives 8-bit weights and 8-bit activations. GGUF is a container from a different runtime lineage. The W-and-A notation beats the brand names because it answers the two questions that matter: how many bits on the weights, and are the activations quantized too.

vLLM picks the kernel by reading the checkpoint, not your command line. AutoAWQConfig and AutoGPTQConfig implement override_quantization_method, which pulls quant_method out of the model's own config. For a pre-quantized checkpoint you do not need --quantization at all, and passing it only asserts: an AWQ checkpoint accepts awq, awq_marlin, auto_awq or marlin, a GPTQ checkpoint gptq, gptq_marlin, auto_gptq or marlin. AutoAWQConfig.get_config_filenames() returns quantize_config.json and quant_config.json, the GPTQ equivalent the first alone.

Open config.json on a shipping 4-bit build:

{
  "quantization_config": {
    "quant_method": "gptq",
    "bits": 4,
    "group_size": 128,
    "desc_act": true,
    "damp_percent": 0.1
  }
}

Activation reordering is on in the checkpoint people actually serve, which decides the accuracy argument below.

The FP8 build declares something structurally different: quant_method: compressed-tensors, format: naive-quantized, ignore: ["lm_head"], and a config_groups block putting weights at num_bits: 8 with strategy: channel and dynamic: false against input activations at strategy: token and dynamic: true. Static per-channel scales on the weights, dynamic per-token scales on the activations: that is W8A8 concretely. The same block carries kv_cache_scheme: null, proof that weight and KV cache quantization are separate decisions with separate accuracy budgets, the cache being set by --kv-cache-dtype and costed in what FP8 and q8_0 KV cache actually cost in accuracy.

GGUF is the odd one out. Since a pull request merged on 12 June 2026 it lives outside vLLM core: gguf is absent from the QuantizationMethods literal in v0.27.1, and serving one needs uv pip install vllm-gguf-plugin first. The claim that GGUF is the CPU and Mac format describes llama.cpp, a different engine with a different operational profile from vLLM; inside vLLM the matrix marks it unsupported on Intel GPU, x86 CPU and Arm CPU.

AWQ is a scaling search, not a rival to FP8: its per-channel scale factors compose with an 8-bit scheme. On llm-compressor's February 2026 run on 8x A100-SXM4-80GB, Meta-Llama-3-8B-Instruct scored 76.42 percent strict-match GSM8K over 1,319 samples with AWQ plus FP8_DYNAMIC against 75.21 percent with AWQ plus FP8_BLOCK, because AWQ's searched scales align with per-channel quantization scales.

vLLM publishes support by SM level: Volta 7.0, Turing 7.5, Ampere 8.0 and 8.6, Ada 8.9, Hopper 9.0. The v0.27.1 table, non-NVIDIA tiers folded into one column:

The asterisk is the footnote that Turing does not support Marlin MXFP4.

Three rows do most of the work. GPTQ is the only one of AWQ, GPTQ and FP8 marked supported on Volta, and the source agrees: AutoGPTQConfig.get_min_capability() returns 60 while the AWQ and FP8 equivalents return 75. llm-compressor FP8 W8A8 is an Ada, Hopper and AMD row only, and W4A8, the combination people ask for by name, has no GPU tier at all.

There is no Blackwell column in v0.27.1, and nothing supports inferring that Blackwell inherits the Hopper rows. Two sentences elsewhere name it: INT4 computation is supported on NVIDIA GPUs with compute capability greater than 8.0 (Ampere, Ada Lovelace, Hopper, Blackwell), and mxfp8 requires SM 100 or newer for W8A8. If you are still choosing silicon rather than a checkpoint for silicon you own, the comparison of RTX PRO 6000, H100 and L40S runs the other direction.

The most repeated wrong sentence in this topic is that FP8 does not work on Ampere. Fp8Config.get_min_capability() returns 75 in v0.27.1, and the documentation is plain: FP8 computation is supported on GPUs with compute capability at or above 8.9 (Ada Lovelace, Hopper), and FP8 models will run from 7.5 (Turing) up as weight-only W8A16, utilizing FP8 Marlin.

Two benefits decouple on older silicon. The memory saving belongs to the file and you get it on a T4; the speedup belongs to the tensor cores and you do not. On an A100 the loader takes the Marlin weight-only path and dequantizes to 16-bit for the matmul, and the FP8 path branches on CUTLASS availability: per-token symmetric activation scales where it exists, per-tensor where it does not. Weight-only is the universal fallback, and mxfp8 degrades the same way below SM 100.

Two numbers deserve correcting. vLLM's FP8 page caps throughput at up to a 1.6x improvement, not 2x, and the 2x there refers to memory. And weight-only quantization can be a throughput regression when memory is not the bottleneck, as a vendor performance lab's Qwen3-8B run on vLLM v0.9.2 over 1,000 ShareGPT prompts shows.

On an RTX 4090 24GB it measured BF16 at 3,869.3 tok/s against 5,653.4 for AWQ (+46.1 percent) and 4,918.98 for GPTQ INT8 (+27.1 percent). On an A100 80GB the ordering inverted: BF16 10,338.25, AWQ 9,611.61, GPTQ INT4 8,129.27. On an H100 80GB, FP8-Static reached 16,452.52 (+26.7 percent) and FP8-Dynamic 15,275.64 (+14.8 percent).

The 4090 gains 46 percent from AWQ because 24 GB is the binding constraint. The A100 loses to every weight-only format because 80 GB is not, and dequantization is pure overhead. Same model, opposite advice.

Four builds of the same Llama-3.1-8B-Instruct, weights measured on disk.

INT8 and FP8 land on the same number: same bytes per quantized weight, same ignored layers. Choosing between them is a hardware and accuracy question, never a capacity one. On a 24 GB card the gap between 5.74 GB and an imagined 4.0 GB is thousands of tokens of KV cache headroom, and that is where peak concurrency turns into a vLLM KV cache OOM.

The evidence base is a study accepted to ACL 2025 that ran over 500,000 evaluations across the Llama-3.1 family on vLLM 0.6.4.post1 (arXiv:2411.02355). FP8 W8A8 came out effectively lossless at all scales, well-tuned INT8 W8A8 at 1 to 3 percent degradation, and INT4 weight-only more competitive than expected, rivaling 8-bit quantization.

Recovery is against the BF16 baseline, so anything above 100 percent is noise. The worst single task was TruthfulQA at 96.88 percent for W4A16-INT on 8B; coding held at 99.9 percent at 8-bit and 98.9 at 4-bit, and long-context RULER at or above 98 percent for every format.

Now the ranking claim. The same paper compared AWQ and GPTQ on Leaderboard V1 and V2, Arena-Hard, HumanEval and MBPP. Academic performance was near-identical, AWQ ahead by 0.23 and 0.35 points on a 0 to 100 scale, while GPTQ led on real-world tasks by 2.9 and 0.8 points: Arena-Hard 52.3 against 49.4 at 8B, 73.1 against 72.3 at 70B. It adopted GPTQ as its primary INT4 method and attributed the difference to configuration, not algorithm: MSE-optimal clipping instead of absmax, and better calibration data than the C4 default.

The perplexity numbers circulating in AWQ's favour come from the AWQ paper's Table 4 (arXiv:2306.00978), usually quoted against the wrong column.

GPTQ-R is GPTQ with activation reordering, and every shipping 4-bit GPTQ checkpoint worth using sets desc_act: true, which is that column. Against it the INT4 delta is 0.01 to 0.05 perplexity. The dramatic version, 6.22 against 5.78, is measured against a reordering-disabled configuration nobody ships. At INT3-g128 on LLaMA-7B: 8.81 for GPTQ, 6.53 for GPTQ-R, 6.35 for AWQ.

The vendor run points the other way at the same magnitude. Qwen3-8B scored BF16 at MMLU 74.78, GSM8K 87.79 and HumanEval 63.41; FP8-Static 74.79, 86.96 and 62.20; INT4-AWQ 73.59, 86.96 and 63.41; INT4-GPTQ 73.26, 86.43 and 62.20. One conclusion fits both: the algorithm name is not what moves the number, the calibration set and the clipping strategy are.

The deployment half of that study inverts the usual advice. Tested on A6000, A100 and H100: in synchronous single-stream serving, W4A16-INT delivered 2 to 3x cost reductions and 1.5 to 2.5x latency improvements at 8B and 70B, rising to 5 to 7x cost reduction at 405B where it also fits on fewer GPUs, while the 8-bit formats showed only modest gains. Under asynchronous continuous batching, W8A8-INT and W8A8-FP yielded the highest throughput, with W4A16-INT still competitive and more efficient under tight latency constraints.

The mechanism is bandwidth against arithmetic. A single decoding stream is bound by bytes of weights crossing the bus per token, so 4-bit weights win even though every matmul dequantizes back to 16-bit. A saturated queue amortises that read across sequences, the bottleneck moves to the matmuls, and only a format that quantizes activations helps there. The same accounting governs speculative decoding, which spends spare arithmetic to save weight reads, so why speculation turns into a slowdown once the queue is full reads this crossover from the other side.

Both sets of numbers carry version stamps: 0.6.4.post1 for the study, v0.9.2 for the vendor lab. Neither describes v0.27.1, and the engine is itself a live variable, which is why the SGLang and vLLM comparison belongs alongside this one.

Quantization looks like a build step until you read the recipes. GPTQ W4A16 in llm-compressor calibrates on 512 samples at 2,048 tokens, AWQ on 256 samples at 512 tokens, and both default to a public instruction dataset that then sits inside weights you run on regulated traffic. vLLM's documentation pushes further: if you have fine-tuned a model, consider using a sample of your training data for calibration.

For a bank or a hospital that sentence has consequences. The calibration set becomes production-adjacent data, inherits its handling rules, and the quantization job has to run where that data is allowed to be: inside the perimeter, on the same air-gapped vLLM footing as serving, with the corpus kept as evidence.

FP8_DYNAMIC is the exception and the reason to prefer it where the silicon allows. Round-to-nearest on weights, dynamic per-token scaling on activations, no dataset, no data-handling question. On Ada or Hopper that is one fewer regulated artefact to govern, at an accuracy cost the ACL 2025 numbers call effectively lossless.

AWQ carries a second cost. Its modifier needs a model-family-specific mapping list (input_layernorm to q/k/v_proj, v_proj to o_proj, post_attention_layernorm to gate/up_proj, up_proj to down_proj for Llama), and for an unlisted architecture you supply your own through the mappings argument.

Then the part that changes your review process. Research published in May 2026 introduced a quantization-conditioned attack that consistently induces malicious behaviour triggered by a broad range of quantization techniques, including AWQ, GPTQ and GGUF I-quants (arXiv:2605.15152). Large outliers cause other weights to be rounded to zero, and an attacker can place them deliberately. On Llama3.1-8B-Instruct the reported jailbreak attack success rate was 95.7 percent after 4-bit GPTQ and 95.0 percent after AWQ, against a full-precision baseline of 2.3 to 33.7 percent, while that same model retained above 90 percent relative utility on MMLU, ARC, HellaSwag, HumanEval and GSM8K in full precision, so the parent passes an ordinary evaluation cleanly. A 2024 result showed the same class of attack against simple zero-shot schemes (arXiv:2405.18137); the 2026 work extends it to data-dependent methods.

Neither is a tracked advisory, so this is procedural, not a patching problem. Evaluate the quantized artefact, not its parent. Quantize in-perimeter from weights you audited rather than downloading someone else's 4-bit build. And record which recipe, which calibration corpus and which library version produced the file you serve, or the artefact is unreproducible and an auditor is entitled to say so. The surrounding controls sit in our LLM models pillar.

Start with the card: Volta means GPTQ, Turing and Ampere mean W4A16 or INT8 W8A8 with FP8 as a memory-only saving through the weight-only Marlin path, Ada and Hopper open FP8 W8A8. Then the batching rule: single-stream picks W4A16, a saturated queue picks W8A8.

Serving a pre-quantized checkpoint needs no flag, because the format is read from the file:

vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

With no pre-quantized file, online quantization converts at load time and needs no calibration data:

vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_tensor

Other online shorthands include fp8_per_block, fp8_per_channel (llm-compressor's FP8_DYNAMIC shape) and mxfp8, and finer control goes through --quantization-config '{"moe":{"activation":"mxfp8"}}'.

To build the checkpoint yourself, llm-compressor is the current path, and a recipe is just a list of modifiers. The AutoAWQ library is deprecated per vLLM's own documentation and its repository archived with a final push in May 2025, so any recipe starting with an AutoAWQ install quotes a dead dependency.

# llm-compressor modifiers, applied with oneshot()

# W4A16 via GPTQ: 512 calibration samples at 2048 tokens
recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])

# W4A16_ASYM via AWQ scale search: 256 calibration samples at 512 tokens
recipe = [
    AWQModifier(duo_scaling="both"),
    QuantizationModifier(ignore=["lm_head"], scheme="W4A16_ASYM", targets=["Linear"]),
]

# FP8 W8A8: no calibration dataset at all
recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
oneshot(model=model, recipe=recipe)

Two GPTQ knobs are documented as the ones to reach for: dampening_frac, where lower values can improve accuracy but can lead to numerical instabilities, and actorder, where actorder="weight" can improve accuracy without added latency. Scheme strings are validated against a fixed preset list (W4A16, W4A16_ASYM, W8A8, FP8, FP8_DYNAMIC, FP8_BLOCK and the generated integer family), so an invented name raises rather than silently degrading.

Evaluate in a separate environment, because llm-compressor and vLLM are documented as possibly not working together:

pip install llmcompressor
# then, in a different environment
pip install vllm "lm-eval[api]>=0.4.12"

lm_eval --model vllm --model_args pretrained=$MODEL,add_bos_token=True \
  --tasks gsm8k --num_fewshot 5 --batch_size auto --limit 250

add_bos_token=True is not optional: quantized models are sensitive to the beginning-of-sequence token and lm_eval does not add one by default, which is how a good checkpoint gets rejected by a bad harness. The published reference on Meta-Llama-3-8B-Instruct-FP8-Dynamic is gsm8k 0.768 plus or minus 0.0268 on both flexible-extract and strict-match.

This week, run that 250-sample gsm8k command against the BF16 model you serve today and write the number down. It takes minutes on a single card, and it is the only thing that turns the quantized replacement from a hope into a decision.