Over the past two decades, we've witnessed two revolutions in applied mathematics and high-dimensional approximation: the rise of sparse reconstruction techniques driven by compressed sensing, and a transformation in data science driven by machine learning with deep neural networks, a.k.a, deep learning. The former seeks to find a compressible representation of a given target function or signal, exploiting structure such as sparsity, parametric smoothness, or low-dimensionality of the solution manifold. The latter seeks to construct a nonlinear approximation from a given dataset, which generalizes well on unseen data points, through a series of compositions of affine and nonlinear mappings. This minisymposium highlights connections between these two topics, with particular attention to recent advances in the theory and algorithms in both approaches, as applied to problems in uncertainty quantification. By bringing together researchers from these two emerging fields, we hope to foster discussion and collaboration on novel theoretical and computational advances in sparse approximation and deep learning, leading to new directions for research.
08:30
Practical Approximation with Deep ReLU Neural Networks
Nick Dexter | Simon Fraser University | Canada
Show details
Authors:
Nick Dexter | Simon Fraser University | Canada
Ben Adcock | Simon Fraser University | Canada
Deep learning (DL) is transforming whole industries as complicated decision-making processes are being automated by neural networks trained on real-world data. Yet as these tools are increasingly being applied to critical problems in medicine, science, and engineering, many important questions about their stability, reliability, and approximation capabilities remain. Such questions include: how many data points are sufficient to train a neural network on simple approximation tasks and how robust are these trained architectures to unknown sources of noise? In this work we seek to quantify the approximation capabilities of deep neural networks (DNNs) both theoretically and numerically. Recent theoretical results show that these architectures allow for the same convergence rates as best-in-class schemes. Our own analysis confirms that DNNs afford the same sample complexity estimates as compressed sensing (CS) on sparse polynomial approximation problems. To explore the approximation capabilities of DNNs, we present numerical experiments on a series of simple tests in high-dimensional function approximation, with comparisons to results achieved with CS on the same problems. In contrast to the theory, our numerical experiments show that standard methods of training often yield DNNs which fail to achieve the theoretical convergence rates, exposing a critical disconnect between the generalization potential of DNNs predicted by the theory and real-world performance of trained DNNs.
09:00
- CANCELED - Nonlinear level-set learning for dimensionality reduction in high-dimensional function approximation
Guannan Zhang | Oak Ridge National Laboratory | United States
Show details
Authors:
Guannan Zhang | Oak Ridge National Laboratory | United States
Jiaxin Zhang | Oak Ridge National Laboratory | United States
Jacob Hinkle | Oak Ridge National Laboratory | United States
We developed a Nonlinear Level-set Learning (NLL) method for dimensionality reduction in high-dimensional function approximation with small data. There are two major challenges in constructing such predictive models: (a) high-dimensional inputs (e.g., many independent design parameters) and (b) small training data, generated by running extremely time-consuming simulations. Thus, reducing the input dimension is critical to alleviate the over-fitting issue caused by data insufficiency. Existing methods, including sliced inverse regression and active subspace approaches, reduce the input dimension by learning a linear coordinate transformation; our main contribution is to extend the transformation approach to a nonlinear regime. Specifically, we exploit reversible networks (RevNets) to learn nonlinear level sets of a high-dimensional function and parameterize its level sets in low-dimensional spaces. A new loss function was designed to utilize samples of the target functions' gradient to encourage the transformed function to be sensitive to only a few transformed coordinates. The NLL approach is demonstrated by applying it to three 2D functions and two 20D functions for showing the improved approximation accuracy with the use of nonlinear transformation, as well as to an 8D composite material design problem for optimizing the buckling-resistance performance of composite shells of rocket inter-stages.
09:30
Deep Neural Networks inspired by quasi-optimal polynomial approximations for parameterized PDEs
Joe Daws | University of Tennessee | United States
Show details
Authors:
Joe Daws | University of Tennessee | United States
Clayton Webster | University of Tennessee | United States
Hoang Tran | Oak Ridge National Laboratory | United States
We show there exists a deep, ReLU activated neural network which achieves the same rate of convergence as quasi-optimal Taylor and Legendre approximations of solutions to a wide class of parameterized elliptic PDEs with finite-dimensional stochastic inputs. The construction of the network provides both an architecture and a set of network parameters capable of approximating the target solution function at least as well as some polynomial approximations. Our numerical experiments indicate that training a network initialized to behave like a polynomial approximation of the solution may yield a better approximation and hence may be used to achieve better estimates of quantities of interest related to that solution.
10:00
Sparse Harmonic Transform : Best s-Term Approximation Guarantees for High-Dimensional Functions in Sublinear-Time
Bosu Choi | University of Texas at Austin | United States
Show details
Authors:
Bosu Choi | University of Texas at Austin | United States
Mark Iwen | Michigan State University | United States
Toni Volkmer | Chemnitz University of Technology | Germany
In this talk we will discuss a sublinear-time compressive sensing algorithm for learning the best s-term approximation of high-dimensional functions which are compressible in Bounded Orthonormal Product Bases (BOPBs). Such functions appear in many applications including, e.g., various Uncertainty Quantification (UQ) problems involving the solution of parametric PDEs that are approximately sparse in Chebyshev or Legendre product bases. We introduce a new variant of the well-known CoSaMP with a sufficiently accurate and efficient support identification procedure satisfying "Support Identification Property (SIP)". Any support identification procedure satisfying the SIP can be used to produce a new CoSaMP variant. As well as the theoretical guarantees, numerical experiments are presented that indicate the developed algorithm allows the solution of sparse approximation problems involving functions contained in the span of fairly general sets of as many as 10^230 orthonormal basis functions.