Machine Learning (ML) has evolved into a core technology in many scientific applications. Solutions often require large labeled datasets to achieve high model accuracy. Unfortunately, this is a major bottleneck for many scientific computing applications, where numerical simulations are very expensive. Training on limited data can lead to significant uncertainties or errors when invoked outside the training space. But the fast execution of ML models once trained also make them ideal for exploring large numbers of runs for Uncertainty Quantification (UQ). Furthermore, many popular ML methods lack the needed mathematical support to prove robustness and reliability to motivate their use in scientific computing and uncertainty quantification UQ applications. This two-part mini-symposium will explore the interplay between ML and UQ, focusing in the following areas: (1) How do we leverage ML successes for scientific computing problems with uncertain inputs? (2) How do we use UQ methods to assess ML predictions and augment them with uncertainty estimates, error bounds, or prediction intervals? Addressing challenges in these areas will lead to greatly improve predictive capabilities. Methods that incorporate mathematical and scientific principles for uncertainty estimates in ML are needed. Literature in statistics can be leveraged for improving the model validation process and advances in UQ and V&V will greatly enhance the mathematical and scientific computing foundations for ML.
16:30
Predictive Uncertainty Estimation in Scientific Machine Learning Models
Ahmad Rushdi | Sandia National Laboratories | United States
Show details
Authors:
Ahmad Rushdi | Sandia National Laboratories | United States
Laura Swiler | Sandia National Laboratories | United States
Presentation Abstract: Machine and deep learning models tend to be overconfident when reporting softmax-based point-estimate predictions, making it hard for the domain expert or decision maker to determine their validity. When dealing with costly numerical simulations or high-stakes decisions, this might be misleading, and a reliable measure of uncertainty is desired. In this talk, we present approaches for data-driven uncertainty estimation and decomposition in scientific machine learning problems using Bayesian neural networks methods such as variational inference, non-Bayesian ones such as ensemble techniques, as well as new hybrid methods. Going beyond objects with distinct features in image analysis problems, we extend these methods to scientific data and highlight how they could be incorporated into augmenting analyst decisions.
17:00
Uncertainty quantification of deep learning predictive models: with application to image-based material property prediction
Jize Zhang | Lawrence Livermore National Laboratory | United States
Show details
Authors:
Jize Zhang | Lawrence Livermore National Laboratory | United States
Bhavya Kailkhura | Lawrence Livermore National Laboratory | United States
T. Yong-Jin Han | Lawrence Livermore National Laboratory | United States
In this work, we investigate into the uncertainty quantification (UQ) of deep learning (DL) prediction models for material properties (e.g., compressive strenght) based on SEM microstructure images. In such context, the ability to quantify the associated confidence or uncertainty for DL prediction would be of utmost importance, because it can be further leveraged to aid important decision makings in cost-effective material discovery and synthesis process. To fulfill such goal, we first compare the performance of Bayesian and non-Bayesian (e.g., ensemble) deep learning uncertainty quantification frameworks for the DL predictive material performance application. The influence of training data quality (including noise and image resolution) and amount on UQ measures are thoroughly examined. The potential use of such UQ measures in active learning and out-of-distribution (OOD) detection tasks will be demonstrated. Finally, a novel post-processing uncertainty calibration technique is also proposed, which ensures that the predictive uncertainties from the learnt DL models can be representative of the ground truth likelihood.
17:30
- CANCELED - Challenges in Bayesian inference via Markov chain Monte Carlo for neural networks
Theodore Papamarkou | Oak Ridge National Laboratory | United States
Show details
Authors:
Theodore Papamarkou | Oak Ridge National Laboratory | United States
Jacob Hinkle | Oak Ridge National Laboratory | United States
Michael Young | Oak Ridge National Laboratory | United States
David Womble | Oak Ridge National Laboratory | United States
Markov chain Monte Carlo (MCMC) methods have been used successfully in Bayesian inference in a wide range of statistical problems. MCMC methods provide the means of conducting uncertainty quantification via posterior density and associated credible interval estimation. However, MCMC techniques have been less popular in machine learning, and deep learning specifically. One of the main reasons for the lack of popularity of Bayesian inference via MCMC for neural networks is the incurring computational complexity. This is not the whole story though. The model structure of neural networks pose other fundamental challenges to MCMC beyond scalability. We identify such challenges in inferring the posteriors of weights and of biases in small neural networks via contemporary geometric and population MCMC sampling. The showcased examples pinpoint some of the fundamental issues that need to be resolved in order to hold some hope to develop effective MCMC sampling methods for neural networks.
18:00
Credibility Processes for Engineering Analyses Using Machine Learning
Nevin Martin | Sandia National Laboratories | United States
Show details
Authors:
Aubrey Eckert | Sandia National Laboratories | United States
Brian Freno | Sandia National Laboratories | United States
Nevin Martin | Sandia National Laboratories | United States
Presentation Abstract: Credibility processes for computational simulations seek to provide a framework for assembling and documenting evidence to ascertain and communicate the believability of predictions that these simulations produce. The verification, validation, and uncertainty quantification communities of practice have developed many methods, tools, and processes for collecting and documenting this credibility evidence for computational analyses. The growing inclusion of machine learning models and methods into the computational simulation workflow requires a reexamination of existing credibility processes to determine where research and development of new methods for generating credibility evidence for these models is needed. Demonstration problems can be used to determine where there may be gaps in both the current processes for assembling credibility evidence and in existing best practices for applying machine learning to engineering analyses. This gap analysis will identify key areas for research and development of methods for assembling credibility evidence around computational simulations using machine learning.