When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

arXiv:2608.24381v2 Announce Type: replace Abstract: In this work, we conduct an empirical study on self-supervised learning (SSL) for tabular data. Our experiments examine limited label regimes (that is, 1%, 5%, 10%, and 20% of the labels) and highlight limitations and opportunities in current tabular SSL. Contrary to what is expected, our mask-and-recover approach shows its largest improvements for datasets that have little to no recorded numeric missingness, whereas it more often leads to degradation on datasets that are characterized by native numeric missingness (post hoc split). Random Forest achieves an average AUC of 0.9015 across the 14 tasks for the 10% labels setting, whereas SSL shows a performance of ~0.8954 (RF: 1 draw; SSL: 3-draw mean). Moreover, the gain of SSL compared to training from scratch exhibits a high variance across tasks and is not significant (1 run; Wilcoxon p = 0.626, Holm-adjusted p = 1.000, for both the 5% and 10% settings). On the other hand, we observe that SSL-pretrained models achieve a higher average AUC when MNAR and MCAR missingness were injected at test-time. In comparison to models trained from scratch, our SSL-pretrained models show an improvement of +0.0245 (MCAR) and +0.0418 AUC (MNAR) under test-time missingness (single seed), with 11 (MCAR; plus one tie) and 8 (MNAR) positive results across the 14 tasks. However, after Holm-Bonferroni correction neither improvement is significant (Holm-adjusted p = 0.118 for MCAR and 0.518 for MNAR). Finally, when comparing mask-and-recover with three other approaches, namely, simplified VIME, SCARF and SubTab implementations using an identical encoder, we observe no significant difference in performance (Holm-adjusted p = 0.459 for VIME, 1.000 for SCARF, 1.000 for SubTab).