DiaVLo: Diagnosing Behaviours of Vision-Language Models
DiaVLo is meant to expose where a vision-language model’s observed behavior diverges from the behavior humans specify.
The framework uses human curation and VLM-generated outputs to build labels for desired and observed behaviors. It also estimates which concepts most strongly steer those behaviors. The authors tested it on several open-source VLMs in classification and generation settings, finding that its behavior labels correlate with performance and add context to the scores. ArXiv · AI/CL/LG's note
The framework uses human curation and VLM-generated outputs to build labels for desired and observed behaviors. It also estimates which concepts most strongly steer those behaviors. The authors tested it on several open-source VLMs in classification and generation settings, finding that its behavior labels correlate with performance and add context to the scores. ArXiv · AI/CL/LG's note
score 5