Can we trust Bayesian uncertainty quantification from Gaussian process priors with squared exponential covariance kernel?

We investigate the frequentist coverage properties of credible sets resulting from Gaussian process priors with squared exponential covariance kernel. First, we show that by selecting the scaling hyperparameter using the maximum marginal likelihood estimator in the (slightly modified) squared exponential covariance kernel, the corresponding L2-credible sets will provide overconfident, misleading uncertainty statements for a large, representative subclass of the functional parameters in the context of the Gaussian white noise model. Then we show that by either blowing up the credible sets with a logarithmic factor or modifying the maximum marginal likelihood estimator with a logarithmic term, one can get reliable uncertainty statements and adaptive size of the credible sets under some additional restriction. Finally, we demonstrate in a numerical study that the derived negative and positive results extend beyond the Gaussian white noise model to the nonparametric regression and classification models for small sample sizes as well. The performance of the squared exponential covariance kernel is also compared to the Mat\'ern covariance kernel.