DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Models
DKL tries to add corpus-specific knowledge to an instruction-tuned model without retraining it on instructions.
The method pre-trains the corresponding base model on the new corpus, then merges those knowledge-infused weights back into the instruction-tuned model. The authors frame it as a lighter alternative to RAFT, PA-RAG, or extended pre-training followed by instruction fine-tuning. In their reported retrieval-failure setting, DKL raises RAG accuracy from 54.17 to 79.26 while using substantially less training data than prior approaches. ArXiv · AI/CL/LG's note
The method pre-trains the corresponding base model on the new corpus, then merges those knowledge-infused weights back into the instruction-tuned model. The authors frame it as a lighter alternative to RAFT, PA-RAG, or extended pre-training followed by instruction fine-tuning. In their reported retrieval-failure setting, DKL raises RAG accuracy from 54.17 to 79.26 while using substantially less training data than prior approaches. ArXiv · AI/CL/LG's note
score 4