Cross-Primitive Consistency in Jev
A lightweight calibrator cut Jev’s inconsistency across query formats by 58.57% and improved accuracy by 2.97%.
The researchers introduced PrimaCons, a benchmark that tests whether Jev makes consistent decisions when the same meaning is expressed through different query formats. Three Jev sizes showed violations across all four benchmark axes. Those discrepancies predicted decision failures with a ROC-AUC of 0.91. The proposed calibrator adds a small residual network to a frozen backbone; the reported gains came under a leave-one-domain-out evaluation.
HF Daily Papers' note
The researchers introduced PrimaCons, a benchmark that tests whether Jev makes consistent decisions when the same meaning is expressed through different query formats. Three Jev sizes showed violations across all four benchmark axes. Those discrepancies predicted decision failures with a ROC-AUC of 0.91. The proposed calibrator adds a small residual network to a frozen backbone; the reported gains came under a leave-one-domain-out evaluation.
HF Daily Papers' note
score 4