Cybersecurity Detection Classification with Reasoning-enabled Language Models
A reasoning-trained triage model beat direct-label LLM classification on real Windows endpoint detections.
The paper trains a chain-of-thought classifier for SOC alert triage using human-labeled endpoint data, prompt optimization, self-training, and reinforcement learning with verifiable rewards. It reports 82.6% test accuracy and, at its high-confidence automated-triage setting, gains of 43.0% benign recall and 18.3% malicious recall over a direct-label LLM baseline. The authors also found that reasoning hurt raw label-token confidence, so they trained a separate calibrator to read the reasoning trace and estimate verdict reliability. An untrained confidence judge failed at that step, while a fine-tuned 30B model outperformed frontier general-purpose models.
ArXiv · AI/CL/LG's note
The paper trains a chain-of-thought classifier for SOC alert triage using human-labeled endpoint data, prompt optimization, self-training, and reinforcement learning with verifiable rewards. It reports 82.6% test accuracy and, at its high-confidence automated-triage setting, gains of 43.0% benign recall and 18.3% malicious recall over a direct-label LLM baseline. The authors also found that reasoning hurt raw label-token confidence, so they trained a separate calibrator to read the reasoning trace and estimate verdict reliability. An untrained confidence judge failed at that step, while a fine-tuned 30B model outperformed frontier general-purpose models.
ArXiv · AI/CL/LG's note
score 4