Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
SiPE adds dependency-parse syntax to positional embeddings without changing Transformer attention or architecture.
The paper tests the syntactic prior across absolute, relative, and rotary positional embeddings for encoders and decoders. It reports architecture-specific best placements: input embeddings for encoders, and a multiplicative link to the relative-position attention term for autoregressive decoders. The authors claim gains of up to 10.3% on SyntaxGym, 9.0% lower perplexity, and up to 8.2% on GLUE versus models without the syntactic supervision. HF Daily Papers' note
The paper tests the syntactic prior across absolute, relative, and rotary positional embeddings for encoders and decoders. It reports architecture-specific best placements: input embeddings for encoders, and a multiplicative link to the relative-position attention term for autoregressive decoders. The authors claim gains of up to 10.3% on SyntaxGym, 9.0% lower perplexity, and up to 8.2% on GLUE versus models without the syntactic supervision. HF Daily Papers' note
score 5