Reproducibility has emerged as an issue in experimental work, and as a consequence, computational work is now being scrutinized for reproducibility. Predating emphasis on reproducibility, computational results have developed methodologies for verification and validation, and more recently with techniques based on probabilistic analysis that are grouped together as Uncertainty Quantification. This minisymposium will be a venue for these ideas and techniques applied to computation.
14:00
The "White Rat" of Numerical Reproducibility
Michael Mascagni | Florida State University | United States
Show details
Author:
Michael Mascagni | Florida State University | United States
We explore an application from the author's work in neuroscience. A code used to investigate neural development modeled 100 neurons with all-to-all excitatory connectivity. We used a simple ordinary differential equation system to model each neuron, and this model was used to produce a paper published in the Journal of Neurophysiology. Later a colleague used our code to continue this work, and found he could not reproduce our results. This lead us to thoroughly investigate this code and we discovered that it offered many different ways to thwart reproducibility.
14:30
Reproducibility and UQ for Two Classes of Geophysical Models
Abani Patra | Tufts University | United States
Show details
Author:
Abani Patra | Tufts University | United States
In this talk we will explore the impact on reproducibility of the use of UQ methodology in the context of two geophysical models. In the first we explore penalized spline based reconstructions of Greenland Ice Sheet surface reconstructions from noisy observation data. In the second we use an ensemble of models and partial inversions for careful model selection among an ensemble of models and predictions based on the best model choice.
15:00
Uncertainty quantification for neuroimaging data analyses
Gregory Kiar | Concordia University | Canada
Show details
Authors:
Tristan Glatard | Concordia University | Canada
Gregory Kiar | Concordia University | Canada
Pablo de Oliveira Castro | University of Versailles | France
Eric Petit | Intel | France
A lack of software reproducibility has become increasingly apparent in the last several years, calling into question the validity of scientific findings affected by published tools. Reproducibility issues may have numerous sources of error, including the underlying numerical stability of algorithms and implementations employed. Various forms of instability have been observed in neuroimaging, including across operating system versions, minor noise injections, and implementation of theoretically equivalent algorithms. We explore the effect of various perturbation methods on a typical neuroimaging pipeline through the use of i) near-epsilon noise injections, ii) Monte Carlo Arithmetic, and iii) varying operating systems to identify the quality and severity of their impact. We demonstrate that even low order computational models such as the connectome estimation pipeline that we used are susceptible to noise. This suggests that stability is a relevant axis upon which tools should be compared, developed, or improved, alongside more commonly considered axes such as accuracy/biological feasibility or performance. The heterogeneity observed across participants clearly illustrates that stability is a property of not just the data or tools independently, but their interaction. Characterization of stability should therefore be evaluated for specific analyses and performed on a representative set of subjects for consideration in subsequent statistical testing. Additionally, identifying how this relationship scales to higher-order models is an exciting next step which will be explored. Finally, the joint application of perturbation methods with post-processing approaches such as bagging or signal normalization may lead to the development of more numerically stable analyses while maintaining sensitivity to meaningful variation.
15:30
Reproducibility, computability, and the scientific method
Peter Coveney | University College London | United Kingdom
Show details
Author:
Peter Coveney | University College London | United Kingdom
We discuss the central importance of reproducibility in the pursuit of scientific knowledge, understanding, and prediction. Unfortunately, at this time it remains true that much computational science is not reproducible, for a number of reasons. Our current work seeks to redress some of these shortcomings through the development of a software toolkit that can provide support for verification, validation and uncertainty quantification, which is applicable and now being used in multiple different domains. In many application domains that one encounters in high performance computing contexts, one must confront both systematic and random errors; in the latter case, one commonly has to deal with chaotic dynamical systems. The question of how to sample these reliably and effectively is integral to the credibility of reported results. We have recently shown that such dynamical systems, when represented on digital computers, manifest a new pathology of the IEEE floating point numbers, which cannot be removed regardless of the level of precision used. In the case of a very simple but representative dynamical system, the generalized Bernoulli map, for which exact results are known, we find that the errors produced by floating point arithmetic can sometimes be catastrophic while, in more generic cases, the errors are of order unity. This indicates that many computational studies of chaotic systems, such as arise in turbulence and molecular dynamics, are likely to contain significant errors which hitherto are unknown to the community of practitioners. I will conclude with some suggestions for how to address this pathology.