Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models
LLMs show measurable sensitivity to “just” and “only,” and instruction tuning appears to sharpen it.
The paper tests how language models handle discourse particles whose effects are subtle and often do not change truth conditions. It finds that both humans and LLMs prefer these particles in context, with stronger sensitivity to “just” than to “only.” In generation tests, models rated outputs containing the particles as more likely given the source context. The authors also identify particle-sensitive units inside models, with asymmetric overlap between “just” and “only” representations. ArXiv · AI/CL/LG's note
The paper tests how language models handle discourse particles whose effects are subtle and often do not change truth conditions. It finds that both humans and LLMs prefer these particles in context, with stronger sensitivity to “just” than to “only.” In generation tests, models rated outputs containing the particles as more likely given the source context. The authors also identify particle-sensitive units inside models, with asymmetric overlap between “just” and “only” representations. ArXiv · AI/CL/LG's note
score 4