01 What happened
Hugging Face has explained how to load GGUF models from the Hub with Transformers’ from_pretrained API by providing a model ID and filename. The packed inference path is MPS-only for now; importing GGUF through dequantization remains a separate option.
02 Key details
- The setup requires an Apple Silicon Mac, a PyTorch version supported by the published ggml-quantization kernel builds (usually one of the two latest PyTorch releases), compatible kernels, and the latest Transformers main branch until the next release.
- The packed loader currently supports Qwen3.5 dense and mixture-of-experts architectures, including compatible Qwen3.8 checkpoints. Hugging Face plans to expand support for other architectures gradually; packed kernels are not available on every device.
- Hugging Face says Transformers came close to llama.cpp across three checkpoints. The measurements are not under identical conditions: the Transformers measurement includes prefill, while llama-bench reports decode-only throughput.
03 Why it matters
Developers with a suitable Apple Silicon Mac can use selected GGUF checkpoints through the familiar Transformers API for experimentation or serving. Packed inference is limited to MPS and the stated Qwen architectures. The source recommends llama.cpp when efficient local inference is the priority.
04 Who it matters to
AI developers.
Original source