AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
AMLworld-Compact cuts AMLworld LLM evaluation targets by 271x and 611x while keeping illicit transactions intact.
The paper proposes an importance-weighted downsampling method that retains every illicit test transaction and samples benign ones by supervised-score strata. Horvitz-Thompson weights are used to estimate full-test confusion counts; recall stays exact, while precision and F1 are estimated as ratios. In held-out tests, the released compact sets had lower F1 RMSE than uniform sampling with HT weights, though conservative intervals stayed broad. The authors also report that seven open-weight LLMs had weighted precision close to a predict-all-illicit baseline under the tested graph prompts. HF Daily Papers' note
The paper proposes an importance-weighted downsampling method that retains every illicit test transaction and samples benign ones by supervised-score strata. Horvitz-Thompson weights are used to estimate full-test confusion counts; recall stays exact, while precision and F1 are estimated as ratios. In held-out tests, the released compact sets had lower F1 RMSE than uniform sampling with HT weights, though conservative intervals stayed broad. The authors also report that seven open-weight LLMs had weighted precision close to a predict-all-illicit baseline under the tested graph prompts. HF Daily Papers' note
score 3