Closing the Context Gap: Activation Alignment for Tabular In-Context Learning
A lightweight activation mapper lets tabular foundation models use smaller contexts while recovering much of the lost accuracy.
The paper trains a linear aligner that pushes a partial-context student’s intermediate activations toward a full-context teacher’s activations. It uses synthetic unlabeled data, needs no GPU, and reportedly converges in seconds to minutes. Tests on 38 TabArena classification datasets with TabPFN-3 and TabFM showed statistically significant gains over unaligned partial-context baselines. In low-data settings, the method recovered nearly half of the teacher’s predictive advantage. HF Daily Papers' note
The paper trains a linear aligner that pushes a partial-context student’s intermediate activations toward a full-context teacher’s activations. It uses synthetic unlabeled data, needs no GPU, and reportedly converges in seconds to minutes. Tests on 38 TabArena classification datasets with TabPFN-3 and TabFM showed statistically significant gains over unaligned partial-context baselines. In low-data settings, the method recovered nearly half of the teacher’s predictive advantage. HF Daily Papers' note
score 4