What an MLX quantized repository does and does not tell you

Denis Ineshin · 2026-10-24 · Read on GitHub

A research memo from mlx-quant-fidelity: what a quantized repo's config and model card record, what they leave out, and what six quality metrics say when three 4-bit Qwen3-0.6B repos are measured anyway

Two 4-bit Qwen3-0.6B repos on the Hugging Face Hub, mlx-community/Qwen3-0.6B-4bit and -4bit-DWQ, differ in size by 36 bytes. A third, -4bit-AWQ, is 9.7 MB larger. Someone choosing among them has the names, the sizes, a config.json and a model card. This memo asks what those records carry and leave out, and what six quality metrics say once the three repos are measured against the unquantized original.

The measurement is one committed comparison report from mlx-quant-fidelity, weight-qwen3-0.6b-ladder.json with its Markdown rendering, shipped in the v0.8.0 release (changelog dated 2026-09-05) and unchanged since. It records mlx 0.31.2, mlx-lm 0.31.3, the device string "Apple M1 Max, 32 GB" and the Hub revision of every repo, but no mlx-quant-fidelity version or commit, macOS version or run date; section 8 discusses that gap under "Build provenance" and lists the other limits. The repo-side evidence is each repo's config.json and README.md at those revisions; mlx-lm's source is cited at v0.31.3, the version the measurement ran, and at v0.24.1, the public tag nearest the version the model cards name. Project README and measurement-principles quotes are pinned to v0.10.0, whose changelog says no committed sample's numbers changed. Nothing was rerun for this write-up. Every number below is read from the committed report, the pinned files or the cited source.

1. The choice in front of the reader

The report scores five of the quantized Qwen3-0.6B repos that mlx-community publishes beside its bf16 repo (bfloat16, a 16-bit floating-point format): three at 4 bits (Qwen3-0.6B-4bit, -4bit-DWQ and -4bit-AWQ), plus -6bit and -8bit. The Hub lists more, at least eight quantized repos, among them a -3bit and two dated DWQ uploads, -4bit-DWQ-05092025 and -4bit-DWQ-053125, none of them measured. The unit here is one DWQ repo at one pinned revision, because an upload under a similar name can hold different weights. The report calls the set a ladder, one rung per repo.

The suffixes name the method each uploader claims:

• Plain 4-bit is round-to-nearest affine quantization. Each group of consecutive weights along a row gets its own scale and bias, stored as two 16-bit values. The group size is the number of weights in a group; 64 is mlx-lm's default.
• DWQ (distilled weight quantization, mlx_lm.dwq) trains a quantized model's scales and biases so that its next-token distribution matches an unquantized teacher's on calibration text.
• AWQ (activation-aware weight quantization, mlx_lm.awq) rescales and clips weights block by block, using activations recorded on calibration text, before quantizing them (awq.py L452–L505). A block whose error does not drop falls back to plain round-to-nearest (L495–L500).
• Dynamic quantization (mlx_lm.dynamic_quant) estimates each layer's sensitivity on calibration text and gives the most sensitive layers more bits, so one model mixes bit widths (dynamic_quant.py L38–L146).

The plain and DWQ weight files differ by 36 bytes on disk (335,450,584 against 335,450,548). Their measured geometry is identical (section 3), so those bytes are an unexplained file-size difference, not a difference in array bytes.

2. What a repo records depends on the tool that wrote it

At their pinned revisions, the plain and DWQ config.json files are byte for byte identical, with the same SHA-256 (15d3ac26…; plain, DWQ). Each has a quantization block and a quantization_config copy holding the same two values, group_size 64 and bits 4, with no method field and no per-module entry. Nothing in them separates a DWQ repo from a plain one.

The AWQ repo's config has a different shape. Its quantization block carries the same two top-level values, then one entry per leaf module (a module with no submodules, such as one linear layer or one norm): 197 dicts of group_size and bits, and 141 false entries. One dict differs from the top level: "model.embed_tokens": {"group_size": 32, "bits": 4}. There is no lm_head entry, because Qwen3-0.6B ties its output head to the token embedding (tie_word_embeddings: true in the reference config). Nor is there a quantization_config copy. This config records its geometry module by module, but it names no method and no calibration data.

The model cards add little. All six README.md files carry the same front matter (library_name: mlx, base_model: Qwen/Qwen3-0.6B, tags: mlx, pipeline_tag: text-generation, license: apache-2.0) and the same sentence: the repo "was converted to MLX format from Qwen/Qwen3-0.6B using mlx-lm version 0.24.0". The DWQ card is that template plus a usage example; the others were read through the version line and usage block. Apart from the repo name in the title, no card names a quantization method or calibration data.

Those shapes follow from what mlx-lm's tools write. The loader reads a per-module entry as a dict of parameters for that module, True for the top-level defaults, or False for no quantization (utils.py L348–L366). At 0.31.3:

• DWQ adds no marker. Unless given an already quantized student, it quantizes a copy of the base model with the quantize_model function mlx_lm.convert uses, at its default mode and with no recipe (dwq.py L368–L383). It trains only the scales and biases (L90–L97) and saves through the shared save (L405–L411), whose only change to the block is the quantization_config copy (utils.py L914–L915).
• AWQ writes a placeholder top level, {"group_size": 64, "bits": 4} under a # dummy comment, whatever the run used, then a dict for each quantized leaf and False for every other, norms and rotary embeddings included (awq.py L513–L530; at v0.24.1, L530–L535). By default it quantizes the token embedding at group 32 (L403–L406, L420–L423; at v0.24.1, L405, L422).
• Dynamic quantization's predicate returns a dict for a sensitive layer, True for an ordinary one and False for one that cannot be quantized (dynamic_quant.py L229–L242); on an unquantized model only the dicts are recorded, under a top level that carries the low bit width (utils.py L815–L835).
• The converter skips a module whose input width is not a multiple of the group size and records nothing (utils.py L823–L827); with no entry and no scales in the weights file, the loader leaves it at full precision (L353–L355).

The AWQ config's shape is consistent with the AWQ script at v0.24.1, the public tag nearest its card's 0.24.0: a per-leaf map with false entries over a group-64 top level and a group-32 embedding, and no quantization_config copy, which that version's save_config does not write (utils.py L473–L494, v0.24.1). At 0.31.3 the same script would also write the copy. The plain and DWQ blocks also lack the mode key that 0.31.3's quantize_model writes (utils.py L814), another sign of an older writer. No 0.31.3 quantizer writes its name into the config, GPTQ included (GPTQ quantizes column by column and adjusts the remaining weights to offset the error, gptq.py L150–L159).

On the read side, mlx-lm's loader rewrites four foreign method strings. When a repo converted from another ecosystem has no native quantization block and its Hugging Face quantization_config names a quant_method of mxfp4 (a 4-bit floating-point format with shared scales), compressed-tensors (a Hugging Face checkpoint format), awq or gptq, load_model replaces the block with a native one that has no method field (utils.py L368–L390), and load returns that rewritten config (L491, L499–L500). Other strings pass through. To see one of those four, a tool has to read config.json from disk.

"Not recorded" is narrower than "undetectable": comparing a repo's scales with plain round-to-nearest quantization of the reference weights could probably tell plain from not plain, though not name the method, and it was not tried. Every weight report from the tool carries a standing note: "Quantization method is not recorded in the repo config … This report describes the geometry of the loaded model, not the method that produced it." The rows below carry repo ids, not method names.

3. Nominal bits and effective bits

The nominal bit width is the one a repo declares in its name and top-level bits. The effective bit width is the loaded model's array bytes times eight, divided by its parameter count before quantization; scales and biases add to the bytes but not to the count; a parameter left unquantized counts on both sides. It is the figure mlx_lm.convert prints (utils.py L196–L215, L847–L848). A uniform 4-bit model at group 64 comes to about 4.5: four bits per weight plus two 16-bit values per 64 weights, 4 + 2·16/64. Since v0.8.0 the weights probe reads geometry and effective bits from the loaded modules rather than the config (CHANGELOG, v0.8.0; probes/weights.py).

All five report zero quantizable modules left at full precision. The plain and DWQ repos match here as they do in their configs. The AWQ row agrees with its config: 196 modules at group 64 and one at group 32, which pays four bits plus two 16-bit values per 32 weights, five bits per weight by the same formula.

That module is the token embedding the AWQ config names, and its size gives a cross-check. Qwen3-0.6B's embedding is 151,936 tokens by 1,024 dimensions, 155,582,464 weights, per the reference config. Half a bit more on each of those weights comes to 9,723,904 bytes. The AWQ repo is 345,174,452 bytes on disk: exactly 9,723,904 more than the DWQ repo, and 9,723,868 more than the plain one, whose extra 36 bytes are the same unexplained gap. The module holds about a quarter of the model's weights, and because the embedding is tied, it is also the output head that produces every next-token score.

A dynamic repo would widen the gap between nominal and effective bits. Its top-level bits is the low width: 4 (--low-bits), with --high-bits 5 and --target-bpw 5.0 at the defaults (dynamic_quant.py L163–L168), so a tool that reads the top level calls a mixed model 4-bit. mlx-quant-fidelity did so before v0.8.0; its headline now shows a mixed 4/5-bit form.

4. Six columns that disagree

Bits and bytes are the cost side. For quality, the report runs each quantized repo and the bf16 reference teacher-forced over the same text: both read the corpus tokens, never their own predictions. It compares their logits, the raw scores over the vocabulary, position by position. The companion memo How to measure what quantization actually costs covers the method and its guards. The text is the first 100 chunks of 512 tokens of the WikiText-2 raw test split, Wikipedia text: 51,200 tokens, 511 scored positions per chunk, 51,100 per repo. The README at v0.10.0 gives the command with all six revisions pinned; running it again would be a new measurement, not a check of this one.

• KL mean and KL median: the Kullback–Leibler divergence KL(P_ref ‖ Q_quant) over the full vocabulary, per position, then averaged or its median taken over the 51,100 positions. Units are nats (natural-log units); lower means closer to the reference.
• KL p99 and KL max: the 99th percentile and the single worst position of that per-position divergence.
• Flip rate: the fraction of positions where the quantized model's most likely next token differs from the reference's.
• Perplexity Δ: perplexity is the exponential of the mean negative log-likelihood of the corpus's actual next tokens; the column is the quantized model's value minus the reference's (29.167).

The first two rows are the pair with identical configs. The DWQ repo is worse on mean KL (0.3539 against 0.2538) and median KL, and better on the other four: p99, max, flip rate and perplexity delta.

The median matters. Had a few very bad positions raised DWQ's mean, the median would barely move, yet it rises from 0.1842 to 0.2966. The DWQ repo sits further from the reference at a typical position and closer at its worst ones. Why the bulk and the tail move apart stays open; the report holds only summaries.

The six columns are not independent witnesses: every one comes from the same logits on the same text. They also answer two different questions. The four KL summaries and the flip rate measure how far the quantized model moves from the reference. Perplexity delta compares how well each model predicts the corpus's actual next tokens, so how it tracks mean KL depends on how the reference distribution aligns with those tokens. The README says of perplexity delta that "it is not independent corroboration" (README, v0.10.0).

5. Ranking has to choose an axis

The compare command ranks repos on a Pareto frontier with two axes. Quality is mean KL alone: "Ranking and domination in compare use mean KLD only" (measurement principles, v0.10.0). Cost is the summed size of the weight files on disk. A repo is dominated when another is no worse on both axes and better on at least one; the frontier is the set nothing dominates.

On the quality axis the DWQ repo is the worse of the pair. It stays on the frontier, with all four others, only through its unexplained 36-byte edge. Rank on p99, max, flip rate or perplexity delta instead, and it would dominate the plain repo. The ranking makes that choice before the reader sees the table.

The choice has a second edge. Mean KL is close to what DWQ trains on, and what DWQ trains on depends on the mlx-lm version. The cards name mlx-lm 0.24.0, but upload_to_hub writes that line at upload time (utils.py L331, v0.24.1; L648–L671, v0.31.3), so it records the uploader's version, not necessarily the one that ran the quantizer. mlx-lm has no public v0.24.0 tag. At v0.24.1, the nearest one, DWQ's loss is kl_div_loss over the full vocabulary, with the student's log-probabilities as inputs and the teacher's as targets, at a temperature of 0.5 (dwq.py L44, L84). It adds 0.1 times a term that matches the student's intermediate-layer activations to the teacher's (L45–L46, L88–L94). By default it trains on 1,024 samples of allenai/tulu-3-sft-mixture (L173, L181–L184).

By 0.31.3 the script had changed: temperature 2.0 (dwq.py L80), no activation term, 2,048 samples (L274), and an opt-in --target-dir mode that keeps the teacher's top 1,024 logits and renormalizes both distributions over them (L59–L60, L111–L114). Its kl_div_loss takes log-probabilities with the teacher's as the target (tuner/losses.py L377–L386). MLX computes sum(exp(targets) * (targets - inputs)) (mlx nn/losses.py L312–L336), and the fused Metal kernel used on Apple GPUs accumulates the same sum (tuner/losses.py L151). At both versions that is KL from teacher to student, the direction the report measures.

In both scripts, then, the ranking axis is a close relative of DWQ's objective: KL to the same kind of teacher, at a different temperature (0.5 or 2.0 against the report's 1), with an added activation term at v0.24.1, and on different text. The report scores WikiText-2, while DWQ's default calibration set is an instruction-tuning mixture. So "DWQ is worse" narrows to this: on WikiText-2, by mean KL, this repo sits further from the reference than the plain one while matching its top token more often. Whether the gap would shrink on text like the calibration set is a hypothesis this evidence does not test. This memo does not claim that the DWQ repo overfits its calibration data.

6. The AWQ repo: best on every column, larger on disk

The AWQ repo is the best of the three 4-bit repos in every column of the section 4 table, with a mean KL of 0.1608. It also costs more: 345.2 MB against 335.5 MB, and 4.632 effective bits per weight against 4.501.

As the output head, the group-32 embedding (section 3) shapes every distribution the six columns score. This evidence cannot separate the block-wise rescaling from the finer-grouped output head as the source of the better scores.

For scale, the 6-bit repo reaches a mean KL of 0.0213 at 484.4 MB, about 7.5 times lower than the AWQ repo's; the 8-bit repo reaches 0.0040 at 633.4 MB. No 4-bit repo comes near the 6-bit rung.

7. Other public measurements

Sam McLeod's KL measurements with mlx-kld (April 2026) report the opposite DWQ result on a different model: on Qwen3.6-35B-A3B, against an 8-bit reference on WikiText-2, the -4bit-DWQ repo's mean KL is about three times lower than plain 4-bit's. A different model and reference, so no contradiction, but one pair is not a verdict on DWQ. Dutta et al., Accuracy is Not All You Need, find KL and "flips" strongly correlated across compression schemes; their flips are answers that change between correct and incorrect on labelled benchmarks, not this memo's per-position top-token disagreement. For contrast, llama.cpp's GGUF format records which calibration dataset an importance-matrix quant used, in a quantize.imatrix.dataset key (PR #6658).

8. What this evidence does not show

• One model, one corpus, one window: Qwen3-0.6B, WikiText-2 prose, 512-token chunks. A second corpus, ideally like a quantizer's calibration data, would test the hypothesis in section 5; none was measured.
• The AWQ attribution: the controlled test, a plain group-64 model with only the embedding at group 32, was not measured.
• The group-32 module's identity comes from the config and the arithmetic; the report counts modules by geometry only.
• Methods and versions: no measured repo is verified to come from the tool its name suggests, or from its defaults. The cards name mlx-lm 0.24.0, the upload-time version; the source cited is 0.24.1, the nearest public tag, and 0.31.3; neither is shown to be the build behind a repo.
• No dynamic repo: section 3's dynamic points rest on source alone.
• One run per repo, so no gap carries an interval.
• Tokenizers: every row carries the warning "Assumes both repos share a tokenizer; only vocab_size (and bos/eos ids) were checked."
• Fidelity is not task quality: five columns measure departure from the bf16 reference on this text and the sixth how well each model predicts it; the report's provisional verdict labels are not used here.
• Build provenance: the report records no mlx-quant-fidelity version or commit, no macOS version and no run date; its fields (the geometry 0.8.0 added, without the dataset_revision 0.10.0 added) place the build between 0.8.0 and 0.10.0.

9. Reading a quantized repo

Read the config and the loaded model before the name. The method suffix is the uploader's claim. What the config adds depends on the tool that wrote it, and none of these three configs records a method or calibration set. Effective bits per weight, read off the loaded model, show what a repo spends.

Look at more than one column: on this pair the six split two against four, and the ranking uses mean KL. Then ask what text a learned quantizer was fit on. Scored on its own calibration text, a quantizer's objective reports how well the fit went; on other text, how well it carried over. For these repos, nothing records whether WikiText-2 is calibration text or other text. How much that matters varies by method: the AWQ paper reports AWQ losing far less than GPTQ when calibration and evaluation text differ. mlx-lm 0.31.3's AWQ, GPTQ and dynamic scripts default to one fixed calibration file (quant/utils.py L8–L26), but a repo does not say whether that default was used.

The KV-cache side is covered in Low-bit KV caches on MLX: what exists and what is missing.

Denis Ineshin · 2026-10-24 · ineshin.space