What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion
Stripping colour and highlighting from policy documents left agents confidently identifying the wrong requirements.
In a test using 76 real policy documents, requirement-recovery micro F1 fell from 0.690 with the authors’ approach to 0.150 with Docling and 0.075 with plain text. The gap held across six answering models from two vendors. Highlight and colour accounted for the measured losses; bold, italic, monospace and strikethrough had near-zero effects. Explicitly identifying the cue’s type for the agent improved performance beyond simply preserving it.
ArXiv · AI/CL/LG's note
In a test using 76 real policy documents, requirement-recovery micro F1 fell from 0.690 with the authors’ approach to 0.150 with Docling and 0.075 with plain text. The gap held across six answering models from two vendors. Highlight and colour accounted for the measured losses; bold, italic, monospace and strikethrough had near-zero effects. Explicitly identifying the cue’s type for the agent improved performance beyond simply preserving it.
ArXiv · AI/CL/LG's note
score 5