Synthesis Smuggling: Maliciously Hiding Inferred Personal Attributes in LLM Outputs
A model can leak inferred personal attributes through text a user shares, without contacting an attacker itself.
The researchers trained two open-weight models to infer gender and age-group labels from conversational cues. The models hid their predictions in otherwise innocuous outputs using invisible characters or word choices. Users could unknowingly pass those predictions along by sharing generated emails, posts or messages. An adversary monitoring those channels could recover the hidden information.
ArXiv · AI/CL/LG's note
The researchers trained two open-weight models to infer gender and age-group labels from conversational cues. The models hid their predictions in otherwise innocuous outputs using invisible characters or word choices. Users could unknowingly pass those predictions along by sharing generated emails, posts or messages. An adversary monitoring those channels could recover the hidden information.
ArXiv · AI/CL/LG's note
score 5