An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations
The paper tests structured audio-caption scoring by deliberately corrupting ground-truth annotations and checking whether the metrics notice.
The authors propose evaluating audio descriptions across five axes: tag sets, free-text descriptions, reasoning, numeric measurements, and spectral profiles. Their framework mixes LLM judges for semantic judgments with deterministic metrics for acoustic deviations. Validation comes from controlled perturbations that add typed, graded errors to AudioCards annotations. The reported result is that the framework separates harmless paraphrase from real semantic or acoustic damage. ArXiv · AI/CL/LG's note
The authors propose evaluating audio descriptions across five axes: tag sets, free-text descriptions, reasoning, numeric measurements, and spectral profiles. Their framework mixes LLM judges for semantic judgments with deterministic metrics for acoustic deviations. Validation comes from controlled perturbations that add typed, graded errors to AudioCards annotations. The reported result is that the framework separates harmless paraphrase from real semantic or acoustic damage. ArXiv · AI/CL/LG's note
score 4