No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation
The paper proposes a way to test LLMs with names that look real but have no found evidence of belonging to a real person.
The authors define “unknown” names as plausible first-last combinations with no indexed full-name evidence or ambiguity signals in a documented validation run. Their PUN protocol combines Wikidata-derived name parts, web-enabled LLM screening, and controlled search revalidation. In a 204-person study, accepted names were judged more name-like than controls, while participants found person evidence in only 3% of cases. The authors release 300 names plus comparison controls. ArXiv · AI/CL/LG's note
The authors define “unknown” names as plausible first-last combinations with no indexed full-name evidence or ambiguity signals in a documented validation run. Their PUN protocol combines Wikidata-derived name parts, web-enabled LLM screening, and controlled search revalidation. In a 204-person study, accepted names were judged more name-like than controls, while participants found person evidence in only 3% of cases. The authors release 300 names plus comparison controls. ArXiv · AI/CL/LG's note
score 4