Megadose The AI curve, compressed.

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

· ArXiv · AI/CL/LG ·
Sharing experts can reduce errors at individual reasoning steps while making the final answer worse.

The paper derives criteria in specified linear regression models that balance variance savings against bias from sharing across roles with different targets. Controlled experiments demonstrate this reversal between local and final-output error. In low-data symbolic tasks, trained Transformers show statistically supported one-step gains, but final-output gains during free-running generation remain statistically unresolved. The authors also show that correct predictions and routing labels alone cannot establish which operations individual experts perform.

ArXiv · AI/CL/LG's note

score 4

Categories: Research