Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
The paper argues that real paragraph structure shows up in how deeply attention compresses, not merely whether it compresses.
Xiang tests hierarchical RoPE by separating paragraph, sentence, and token positions while keeping the token sequence fixed. Cross-paragraph attention compresses across every corpus, but random density-matched paragraph labels compress too. The claimed signal is that true paragraph boundaries produce deeper, corpus-dependent compression, unlike the control. Embedding-based coherence comes closest to matching that pattern, but none of the tested corpus measures fully explains it. HF Daily Papers' note
Xiang tests hierarchical RoPE by separating paragraph, sentence, and token positions while keeping the token sequence fixed. Cross-paragraph attention compresses across every corpus, but random density-matched paragraph labels compress too. The claimed signal is that true paragraph boundaries produce deeper, corpus-dependent compression, unlike the control. Embedding-based coherence comes closest to matching that pattern, but none of the tested corpus measures fully explains it. HF Daily Papers' note
score 4