Transformers can now run llama.cpp's GGUF quants

Hugging Face has bridged its own library and llama.cpp's file format. A post published on 22 September by Marc Sun, Arthur Zucker and Lysandre Debut explains that transformers can now load GGUF checkpoints through the same from_pretrained call it uses for everything else, reusing llama.cpp's ggml kernels through Hugging Face's kernels package. The first target is local inference on Apple Silicon, beginning with the Qwen3.5 family.

A 4B model in 2.7 GB instead of 8.4 GB

GGUF packs weights, tokenizer information and an optional chat template into one file, and its quantization levels trade precision for memory. Using Unsloth's Qwen3.5-4B as the worked example, Hugging Face lists BF16 at 8.42 GB, Q6_K at 3.53 GB, Q5_K_M at 3.14 GB and Q4_K_M at 2.74 GB. The advice is to start at Q4_K_M and step up if the machine has room, because the quality trade-off depends on the model and the task.

Two arguments and a file name

Loading one is deliberately unglamorous: hand over a Hub repository and the .gguf file name, and transformers reads the checkpoint. When the weights stay packed on Metal, the library picks up the compatible ggml and Metal kernels on its own and uses ggml-attn for attention; if that kernel cannot be fetched it falls back to sdpa with a warning. The requirements are an Apple Silicon Mac, one of the two most recent PyTorch releases, the development version of transformers and a matching copy of kernels.

An OpenAI-compatible door into the same checkpoint

The same file can be served rather than scripted. transformers serve takes the repository and the quantization after a colon and exposes an OpenAI-compatible endpoint, so a desktop client such as Jan or Pi can talk to a local model without any Python. For models whose chat template supports thinking, a reasoning flag turns that behaviour on, off or to the template's own default.

Close to llama.cpp, with a caveat attached

Hugging Face benchmarked three checkpoints (a small dense model, a larger dense model and a mixture-of-experts model) on a MacBook Pro M2 Max with 32 GB of unified memory, macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0, against llama-bench build b10200. It reports performance close to llama.cpp across all three, with two changes to the generation loop, dropping an unnecessary attention mask early and deferring the stopping check, helping every transformers model rather than only GGUF. The caveat is printed on the page: the transformers figures include prefill while llama-bench reports decode-only throughput, so the two columns are not measured under identical conditions.

What it does not do yet

The packed path is MPS-only for now. Padding and batching still need work, since unpadded input benefits from the mask change but padded batches cannot take the shortcut. Architecture coverage is limited to the Qwen3.5 dense and mixture-of-experts families, including compatible Qwen3.8 checkpoints. Hugging Face still recommends llama.cpp as the engine for efficient local inference, positioning this as a way to work with the same files inside Python and PyTorch: inspect intermediate activations, amend a forward pass, evaluate a quantized checkpoint with the tools already to hand.

Our opinion

GGUF's value to Hugging Face is not that a laptop gets faster; it is that the format stops being a wall. Quantized checkpoints used to be something you ran and could not open, and loading one with from_pretrained changes the kind of question a developer can ask: where did this quantization cost me, and on which layer. The comparison itself deserves the scepticism the authors invite, because prefill sits in one column and not the other, so treat close to llama.cpp as a claim to reproduce rather than a result to quote. The scoping is narrow, Metal first and Qwen3.5 first, which is a respectable way to ship an integration without pretending the whole Hub already works.