Deep learning techniques are becoming the center of attention across many scientific disciplines. Many predictive tasks are currently being tackled using over-parameterized, black-box discriminative models such as deep neural networks, in which interpretability and robustness is often sacrificed in favor of flexibility in representation and scalability in computation. Such models have yielded remarkable results in data-rich domains, yet their effectiveness in data-scarce and risk-sensitive tasks still remains questionable, primarily due to open challenges in statistical inference and uncertainty quantification. This mini-symposium invites contributions on uncertainty quantification methods for deep learning and their application in the physical and engineering sciences. Topics include (but are not limited to) Bayesian neural networks, deep generative models, posterior inference techniques, and applications to forward/inverse problems, active learning, Bayesian optimization and reinforcement learning.
16:30
Sampling the posterior of Bayesian neural networks, with neural networks
Yibo Yang | University of Pennsylvania | United States
Show details
Authors:
Yibo Yang | University of Pennsylvania | United States
Paris Perdikaris | University of Pennsylvania | United States
In this work we explore the potential of deep generative models for generating approximate samples from complex high-dimensional distributions, such as the posterior of deep neural networks. Specifically, we formulate a variational inference framework that iteratively refines the choice of an appropriate proposal distribution for importance sampling that is parametrized by a deep generative model. The latter aims to learn the map between samples generated in tractable and low-dimensional latent space to samples originating from the target unnormalized distribution.
The proposed framework can be easily adapted into different application scenarios and we present numerical studies that showcase its effectiveness in sampling high-dimensional posterior distributions arising in Bayesian neural networks. Moreover, the quality of the resulting uncertainty estimates is assessed in several decision-making tasks including contextual bandits problems and Bayesian optimization of non-convex functions.
17:00
Refining the variational posterior through iterative optimization
Marton Havasi | University of Cambridge | United Kingdom
Show details
Authors:
José Miguel Hernández Lobato | University of Cambridge | United Kingdom
Marton Havasi | University of Cambridge | United Kingdom
Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks. A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive. In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it. Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families. We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks. In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO. We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10. Our method is general and could have applications in various uncertainty quantification problems.
17:30
Uncertainties in Attention-based Koopman Embeddings
Ludger Paehler | Technical University of Munich | Germany
Show details
Authors:
Ludger Paehler | Technical University of Munich | Germany
Steven L Brunton | University of Washington | United States
Nikolaus Adams | Technical University of Munich | Germany
The Koopman operator provides great promise for the embedding of nonlinear dynamics on a manifold where the dynamics are approximately linear. When the, most often elusive, Koopman operator is known its modes enable a linear estimation of the time evolution on a reduced number of degrees of freedom. As a result it can significantly accelerate various uncertainty quantification tasks associated with uncertainty propagation and stochastic control of high-dimensional dynamical systems. The first proposal for this were autoencoders combined with a separate Koopman eigenfunction encoding network on the latent space. But the Autoencoder framework introduces a number of constraints, the most important of which is the fixed size of the latent space resulting in a performance decline for complex dynamical system. Building on advances in natural language processing, we propose to combine attention-based networks with the Koopman network approach. A proposal, which ends the use of a single, fixed size latent space in favour of multiple, dynamically size-adjusted latent spaces to increase the flexibility and representational capability of the network. Of utmost importance to this approach is a quantification of the uncertainties in the selection of the Koopman modes. This is done with a Sobol sensitivity like approach in which sensitivites up to the 2nd order are analyzed. We demonstrate feasibility and extension to a broader class of problems on 3D Taylor-Green vortex-flow transitions.
18:00
Integration of adversarial autoencoders with residual dense convolutional networks for estimation of non-Gaussian conductivities in solute transport modeling
Nicholas Zabaras | University of Notre Dame | United States
Show details
Authors:
Shaoxing Mo | Nanjing University | China
Nicholas Zabaras | University of Notre Dame | United States
Inverse modeling involving characterization of a non-Gaussian conductivity field in subsurface flow and transport constitutes a challenging problem. This is mainly due to the non-Gaussian property and the fact that many repeated evaluations of the forward model are often required. In this study, we develop a convolutional adversarial autoencoder (CAAE) to parameterize the non-Gaussian conductivity fields with heterogeneous conductivity within each facies using a low-dimensional latent representation. In addition, a deep residual dense convolutional network (DRDCN) is proposed for surrogate modeling of solute transport models with high-dimensional and highly-complex mappings. The two networks are both based on a multilevel residual learning architecture called residual-in-residual dense block. The multilevel residual learning strategy and the dense connection structure ease the training of deep networks, enabling us to efficiently build deeper networks that have an essentially increased capacity for approximating mappings of very high-complexity. The CCAE and DRDCN networks are incorporated into an iterative local updating ensemble smoother to formulate an inversion framework. The integrated method is demonstrated using 2-D and 3-D solute transport models with non-Gaussian conductivity fields. The obtained results indicate that the CAAE is a robust parameterization method for non-Gaussian conductivity fields with different heterogeneity patterns.