Occam's Razor is Only as Sharp as Your ELBO

Researchers at arXiv (2604.25984) expose a critical failure mode in ELBO-based hyperparameter learning: Gaussian approximate posteriors with reduced-rank covariance matrices can drive overfitting in over-parameterized regression, not just underfitting as previously documented. Counterintuitively, the true marginal likelihood sometimes selects the overfit solution while the ELBO rejects it, revealing asymmetric failure modes between the two objectives. Bayesian ML engineers scaling variational inference to large models must audit their rank assumptions carefully, as tractability shortcuts directly corrupt model selection guarantees that practitioners typically take for granted.