LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics
LexFlip tests whether legal-meaning metrics notice force reversals when almost all the words stay the same.
The paper releases 373 minimal perturbations of Quebec statutory French that preserve about 0.93 of tokens while reversing legal force. Seven embedding and BERTScore metrics barely move on those edits, using only 0.022 to 0.039 of their identical-to-unrelated range. Bidirectional NLI moves much more, at 0.670, though the paper says the usual identical-pair check would reject that family. On FrJudge, a bare length feature beats every semantic metric against a measured human ceiling of r=0.597. ArXiv · AI/CL/LG's note
The paper releases 373 minimal perturbations of Quebec statutory French that preserve about 0.93 of tokens while reversing legal force. Seven embedding and BERTScore metrics barely move on those edits, using only 0.022 to 0.039 of their identical-to-unrelated range. Bidirectional NLI moves much more, at 0.670, though the paper says the usual identical-pair check would reject that family. On FrJudge, a bare length feature beats every semantic metric against a measured human ceiling of r=0.597. ArXiv · AI/CL/LG's note
score 4