Understanding the Surprising Generalization Properties of Tabular Foundation Models
A single real table can be enough to produce strong transfer in tabular foundation models.
The paper argues that useful pre-training depends more on the number and quality of tasks a table implies than on sheer dataset volume. It finds that high-feature tables tend to transfer better, while dataset-level filtering or deduplication does not improve results. The authors frame tabular in-context generalization as largely retrieval-based: models succeed by finding relevant examples in context and aggregating them well. ArXiv · AI/CL/LG's note
The paper argues that useful pre-training depends more on the number and quality of tasks a table implies than on sheer dataset volume. It finds that high-feature tables tend to transfer better, while dataset-level filtering or deduplication does not improve results. The authors frame tabular in-context generalization as largely retrieval-based: models succeed by finding relevant examples in context and aggregating them well. ArXiv · AI/CL/LG's note
score 4