Self-supervision drives representational convergence in medical foundation models more than clinical supervision
The strongest shared geometry came from matching self-supervised objectives, not labels or model size.
The paper tests 18 image encoders and 7 text encoders across medical modalities, including more than 650,000 chest radiographs. Matched self-supervised encoders aligned far more than label-supervised or image-text models, and convergence did not increase with scale or capability. The shared structure stayed within imaging, failed to match clinical language, and did not mirror radiologists’ similarity judgments. Even so, linear classifiers transferred across encoders and to five held-out hospitals at about 85% of within-encoder performance. ArXiv · AI/CL/LG's note
The paper tests 18 image encoders and 7 text encoders across medical modalities, including more than 650,000 chest radiographs. Matched self-supervised encoders aligned far more than label-supervised or image-text models, and convergence did not increase with scale or capability. The shared structure stayed within imaging, failed to match clinical language, and did not mirror radiologists’ similarity judgments. Even so, linear classifiers transferred across encoders and to five held-out hospitals at about 85% of within-encoder performance. ArXiv · AI/CL/LG's note
score 4