When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs
The paper argues that quantized medical LLMs can keep choosing the right answer while losing the rationale evidence that made the answer trustworthy.
The authors test post-training quantization on medical multiple-choice QA, where explanations are expected to support the selected option. Their method adds an explanation-aware objective that uses full-precision teacher rationales to preserve evidence tokens and evidence-conditioned answer behavior. In their OSTQuant W4A4KV4 experiments across four 7B-8B models and three medical QA datasets, the baseline can preserve accuracy while weakening answer-supporting rationales. The proposed objective is aimed at preserving the full-precision model’s reasoning support, not raising gold-label accuracy. ArXiv · AI/CL/LG's note
The authors test post-training quantization on medical multiple-choice QA, where explanations are expected to support the selected option. Their method adds an explanation-aware objective that uses full-precision teacher rationales to preserve evidence tokens and evidence-conditioned answer behavior. In their OSTQuant W4A4KV4 experiments across four 7B-8B models and three medical QA datasets, the baseline can preserve accuracy while weakening answer-supporting rationales. The proposed objective is aimed at preserving the full-precision model’s reasoning support, not raising gold-label accuracy. ArXiv · AI/CL/LG's note
score 4