Demystifying Agent Skills: Why They Work-Until They Don't
Skills mostly help agents by turning messy prior runs into procedural anchors, not by adding missing facts.
The paper reports controlled experiments across benchmarks, harnesses, and LLMs, with 8,135 trial records normalized and 238 valid labels retained from open coding. Skills beat Workflow Memory by 6.06 points in matched comparisons, and procedural anchoring explains 65.7% of skill cases versus 4.5% for explicit knowledge injection. Retrieval is the weak point: as skill pools grow from 5 to 100, actual-use precision drops from 29.6% to 3.3%. The authors say skills fail when assumptions are brittle, contexts do not fit, or the agent does not adapt enough. HF Daily Papers' note
The paper reports controlled experiments across benchmarks, harnesses, and LLMs, with 8,135 trial records normalized and 238 valid labels retained from open coding. Skills beat Workflow Memory by 6.06 points in matched comparisons, and procedural anchoring explains 65.7% of skill cases versus 4.5% for explicit knowledge injection. Retrieval is the weak point: as skill pools grow from 5 to 100, actual-use precision drops from 29.6% to 3.3%. The authors say skills fail when assumptions are brittle, contexts do not fit, or the agent does not adapt enough. HF Daily Papers' note
score 5