Huawei releases MACaQ, its internal quantizer for local inference
MACaQ claims it can quantize a 122-billion-parameter model in 64 minutes using 32.5 GB of device memory.
The Apache-2.0 toolkit covers text, image, video and audio models through llama.cpp, vLLM, Diffusers and PyTorch. Text workflows support 2-, 3- and 4-bit GGUF, while other backends offer 4-bit weights and activations. Its MACaQ-R refinement preserves native GGUF formats and block scales, so the resulting files run in standard llama.cpp. For Qwen3.8-27B in vLLM, the repository reports 2.33× prefill and 2.58× decode speed versus BF16, with perplexity rising from 2.8463 to 2.8994.
HN · Inference's note
The Apache-2.0 toolkit covers text, image, video and audio models through llama.cpp, vLLM, Diffusers and PyTorch. Text workflows support 2-, 3- and 4-bit GGUF, while other backends offer 4-bit weights and activations. Its MACaQ-R refinement preserves native GGUF formats and block scales, so the resulting files run in standard llama.cpp. For Qwen3.8-27B in vLLM, the repository reports 2.33× prefill and 2.58× decode speed versus BF16, with perplexity rising from 2.8463 to 2.8994.
HN · Inference's note
score 5