Evaluating Quantized Language Model Inference a Critical Review of Quality Memory and Workload Tradeoffs

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Quantization reduces the numerical precision of large language model weights, activations, or key-value caches, but its practical value depends on the interaction among numerical error, workload, and execution hardware. This review compares representative methods through those three dimensions. A purposive literature synthesis covers integer inference foundations, outlier-aware mixed precision, second-order weight reconstruction, activation-aware scaling, rotations, cache quantization, and training with low-bit representations. Rather than pool incompatible benchmark results, the review distinguishes the object being quantized, the required preparation, the retained precision, and the implementation needed for a benefit. An analytical memory model separates compressed weights from quantization metadata, caches, and runtime workspace. A recomputable worked example and concurrency sensitivity analysis show why a fourfold reduction in nominal weight bytes does not imply a fourfold reduction in total serving memory. The paper proposes a reproducible comparison protocol that separates prefill from decoding and evaluates quality, latency distributions, throughput, peak memory, and amortized preparation cost under a common workload. It also distinguishes quantized fine-tuning from inference optimization and native low-bit training from post-training conversion. No new model benchmark, speedup, or accuracy result is reported. The central conclusion is that quantization should be selected against an application-specific operating envelope, with quality constraints and hardware measurements, rather than ranked by bit width alone.