Research Papers

A Linear Regression Framework for the Verification of Bayesian Model Calibration Algorithms

[+] Author and Article Information
Jerry A. McMahan, Jr.

Digimarc Corporation,
Beaverton, OR 97008
e-mail: jerry.mcmahan@digimarc.com

Brian J. Williams

Statistical Sciences Group (CCS-6),
Los Alamos National Laboratory,
Los Alamos, NM 87545
e-mail: brianw@lanl.gov

Ralph C. Smith

Department of Mathematics,
North Carolina State University,
Raleigh, NC 27695
e-mail: rsmith@ncsu.edu

Nicholas Malaya

Department of Mechanical Engineering,
The University of Texas at Austin,
Austin, TX 78712
e-mail: nick@ices.utexas.edu

Manuscript received November 20, 2016; final manuscript received August 14, 2017; published online September 12, 2017. Editor: Ashley F. Emery. The United States Government retains, and by accepting the article for publication, the publisher acknowledges that the United States Government retains, a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for United States government purposes.

J. Verif. Valid. Uncert 2(2), 021006 (Sep 12, 2017) (14 pages) Paper No: VVUQ-16-1031; doi: 10.1115/1.4037705 History: Received November 20, 2016; Revised August 14, 2017

We describe a framework for the verification of Bayesian model calibration routines. The framework is based on linear regression and can be configured to verify calibration to data with a range of observation error characteristics. The framework is designed for efficient implementation and is suitable for verifying code intended for large-scale problems. We propose an approach for using the framework to verify Markov chain Monte Carlo (MCMC) software by combining it with a nonparametric test for distribution equality based on the energy statistic. Our matlab-based reference implementation of the framework is shown to correctly distinguish between output obtained from correctly and incorrectly implemented MCMC routines. Since correctness of output from an MCMC software depends on choosing settings appropriate for the problem-of-interest, our framework can potentially be used for verifying such settings.

Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.


Tiesler, H. , Kirby, R. M. , Xiu, D. , and Preusser, T. , 2012, “ Stochastic Collocation for Optimal Control Problems With Stochastic PDE Constraints,” SIAM J. Control Optim., 50(5), pp. 2659–2682. [CrossRef]
Hays, J. , Sandu, A. , Sandu, C. , and Hong, D. , 2011, “ Motion Planning of Uncertain Fully-Actuated Dynamical Systems: A Forward Dynamics Formulation,” ASME Paper No. DETC2011-48233.
Chauvieère, C. , Hesthaven, J. S. , and Lurati, L. , 2006, “ Computational Modeling of Uncertainty in Time-Domain Electromagnetics,” SIAM J. Sci. Comput., 28(2), pp. 751–775. [CrossRef]
Xiu, D. , and Shen, J. , 2007, “ An Efficient Spectral Method for Acoustic Scattering From Rough Surfaces,” Commun. Comput. Phys., 2(1), pp. 54–72. http://www.global-sci.com/freedownload/v2_54.pdf
Bui-Thanh, T. , and Ghattas, O. , 2014, “ An Analysis of Infinite Dimensional Bayesian Inverse Shape Acoustic Scattering and Its Numerical Approximation,” SIAM J. Uncertainty Quantif., 2(1), pp. 203–222. [CrossRef]
Hu, Z. , Smith, R. C. , Burch, N. , Hays, M. , and Oates, W. S. , 2014, “ A Modeling and Uncertainty Quantification Framework for a Flexible Structure With Macrofiber Composite Actuators Operating in Hysteretic Regimes,” J. Intell. Mater. Syst. Struct., 25(2), pp. 204–228. [CrossRef]
Crews, J. H. , McMahan, J. A. , Smith, R. C. , and Hannen, J. C. , 2013, “ Quantification of Parameter Uncertainty for Robust Control of Shape Memory Alloy Bending Actuators,” Smart Mater. Struct., 22(11), p. 115021. [CrossRef]
Najm, H. N. , 2009, “ Uncertainty Quantification and Polynomial Chaos Techniques in Computational Fluid Dynamics,” Annu. Rev. Fluid Mech., 41(1), pp. 35–52. [CrossRef]
Yu, Y. , Zhao, M. , Lee, T. , Pestieau, N. , Bo, W. , Glimm, J. , and Grove, J. , 2006, “ Uncertainty Quantification for Chaotic Computational Fluid Dynamics,” J. Comput. Phys., 217(1), pp. 200–216. [CrossRef]
Knio, O. , and Maître, O. L. , 2006, “ Uncertainty Propagation in CFD Using Polynomial Chaos Decomposition,” Fluid Dyn. Res., 38(9), pp. 616–640. [CrossRef]
Xiu, D. , and Karniadakis, G. E. , 2003, “ Modeling Uncertainty in Flow Simulations Via Generalized Polynomial Chaos,” J. Comput. Phys., 187(1), pp. 137–167. [CrossRef]
Cui, T. , Marzouk, Y. M. , and Willcox, K. E. , 2014, “ Data-Driven Model Reduction for the Bayesian Solution of Inverse Problems,” Int. J. Numer. Methods Eng., 102(5), pp. 966–990. [CrossRef]
Allaire, D. , and Willcox, K. , 2010, “ Surrogate Modeling for Uncertainty Assessment With Application to Aviation Environmental System Models,” AIAA J., 48(8), pp. 1791–1803. [CrossRef]
Pettit, C. L. , 2004, “ Uncertainty Quantification in Aeroelasticity: Recent Results and Research Challenges,” J. Aircr., 41(5), pp. 1217–1229. [CrossRef]
Green, L. L. , Lin, H.-Z. , and Khalessi, M. R. , 2002, “ Probabilistic Methods for Uncertainty Propagation Applied to Aircraft Design,” AIAA Paper No. 2002-3140.
Hollinger, D. Y. , and Richardson, A. D. , 2005, “ Uncertainty in Eddy Covariance Measurements and Its Application to Physiological Models,” Tree Physiol., 25(7), pp. 873–885. [CrossRef] [PubMed]
Chen, P. , Quarteroni, A. , and Rozza, G. , 2013, “ Simulation-Based Uncertainty Quantification of Human Arterial Network Hemodynamics,” Int. J. Numer. Methods Biomed. Eng., 29(6), pp. 698–721. [CrossRef]
Estep, D. , and Neckels, D. , 2006, “ Fast and Reliable Methods for Determining the Evolution of Uncertain Parameters in Differential Equations,” J. Comput. Phys., 213(2), pp. 530–556. [CrossRef]
Banks, H. T. , Hu, S. , Jang, T. , and Kwon, H.-D. , 2012, “ Modeling and Optimal Control of Immune Response of Renal Transplant Recipients,” J. Biol. Dyn., 6(2), pp. 539–567. [CrossRef] [PubMed]
Floris, F. , Bush, M. , Cuypers, M. , Roggero, F. , and Syversveen, A. R. , 2001, “ Methods for Quantifying the Uncertainty of Production Forecasts: A Comparative Study,” Pet. Geosci., 7(S), pp. S87–S96. [CrossRef]
Kovscek, A. , and Wang, Y. , 2005, “ Geologic Storage of Carbon Dioxide and Enhanced Oil Recovery. I. Uncertainty Quantification Employing a Streamline Based Proxy for Reservoir Flow Simulation,” Energy Convers. Manage., 46(11–12), pp. 1920–1940. [CrossRef]
Bui-Thanh, T. , Ghattas, O. , Martin, J. , and Stadler, G. , 2013, “ A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems—Part I: The Linearized Case, With Application to Global Seismic Inversion,” SIAM J. Sci. Comput., 35(6), pp. A2494–A2523. [CrossRef]
Reed, H. , and Hoppe, W. , 2016, “ A Model-Based, Bayesian Characterization of Subsurface Corrosion Parameters in Composite Multi-Layered Structures,” AIP Conf. Proc., 1706(1), p. 120010.
McMahan, J. A. , and Criner, A. K. , 2016, “ Statistical Flaw Characterization Through Bayesian Shape Inversion From Scattered Wave Observations,” AIP Conf. Proc., 1706(1), p. 130005.
McMahan, J. A. , Aldrin, J. C. , Shell, E. , and Oneida, E. , 2017, “ Bayesian Flaw Characterization From Eddy Current Measurements With Grain Noise,” AIP Conf. Proc., 1806(1), p. 110014.
Ghanem, R., Higdon, D., and Owhadi, H. , eds., 2017, Handbook of Uncertainty Quantification, Springer International Publishing, Cham, Switzerland.
Smith, R. C. , 2013, Uncertainty Quantification: Theory, Implementation, and Applications, Vol. 12, SIAM, Philadelphia, PA.
Sullivan, T. J. , 2015, Introduction to Uncertainty Quantification (Texts in Applied Mathematics), 1st ed., Vol. 63, Springer International Publishing, Cham, Switzerland. [CrossRef]
Calvetti, D. , and Somersalo, E. , 2007, An Introduction to Bayesian Scientific Computing: Ten Lectures on Subjective Computing, Vol. 2, Springer Science & Business Media, New York.
Tarantola, A. , 2005, Inverse Problem Theory and Methods for Model Parameter Estimation, SIAM, Philadelphia, PA. [CrossRef]
Stuart, A. M. , 2010, “ Inverse Problems: A Bayesian Perspective,” Acta Numer., 19, pp. 451–559. [CrossRef]
Roache, P. J. , 1998, Verification and Validation in Computational Science and Engineering, Vol. 895, Hermosa, Albuquerque, NM.
Roy, C. J. , 2005, “ Review of Code and Solution Verification Procedures for Computational Simulation,” J. Comput. Phys., 205(1), pp. 131–156. [CrossRef]
Oberkampf, W. L. , and Trucano, T. G. , 2002, “ Verification and Validation in Computational Fluid Dynamics,” Prog. Aerosp. Sci., 38(3), pp. 209–272. [CrossRef]
Roache, P. J. , 2002, “ Code Verification by the Method of Manufactured Solutions,” ASME J. Fluids Eng., 124(1), pp. 4–10. [CrossRef]
Malaya, N. , Estacio-Hiroms, K. C. , Stogner, R. H. , Schulz, K. W. , Bauman, P. T. , and Carey, G. F. , 2012, “ MASA: A Library for Verification Using Manufactured and Analytical Solutions,” Eng. Comput., 29(4), pp. 487–496.
Raju, A. , Roy, C. , and Hopkins, M. , 2005, “ On the Generation of Exact Solutions Using the Method of Nearby Problems,” AIAA Paper No. 2005-684.
Adams, B. , Hooper, R. W. , Lewis, A. , McMahan, J. A., Jr. , Smith, R. C. , Swiler, L. P. , and Williams, B. J. , 2014, “ User Guidelines and Best Practices for CASL VUQ Analysis Using DAKOTA,” Sandia National Laboratory, Albuquerque, NM, Technical Report No. 2014-2864. https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/SAND-CaslDakotaManual.pdf
Diaconis, P. , 2009, “ The Markov Chain Monte Carlo Revolution,” Bull. Am. Math. Soc., 46(2), pp. 179–205. [CrossRef]
Chib, S. , and Greenberg, E. , 1995, “ Understanding the Metropolis-Hastings Algorithm,” Am. Stat., 49(4), pp. 327–335.
Haario, H. , Laine, M. , Mira, A. , and Saksman, E. , 2006, “ DRAM: Efficient Adaptive MCMC,” Stat. Comput., 6(4), pp. 339–354. [CrossRef]
Solonen, A. , Ollinaho, P. , Laine, M. , Haario, H. , Tamminen, J. , and Järvinen, H. , 2012, “ Efficient MCMC for Climate Model Parameter Estimation: Parallel Adaptive Chains and Early Rejection,” Bayesian Anal., 7(3), pp. 715–736. [CrossRef]
Neal, R. M. , 2011, “ MCMC Using Hamiltonian Dynamics,” Handbook Markov Chain Monte Carlo, CRC Press, Boca Raton, FL. [CrossRef]
Székely, G. J. , and Rizzo, M. L. , 2004, “ Testing for Equal Distributions in High Dimension,” InterStat, 5, pp. 1–6. https://pdfs.semanticscholar.org/ad5e/91905a85d6f671c04a67779fd1377e86d199.pdf
Gregory, K. B. , Carroll, R. J. , Baladandayuthapani, V. , and Lahiri, S. N. , 2015, “ A Two-Sample Test for Equality of Means in High Dimension,” J. Am. Stat. Assoc., 110(510), pp. 837–849. [CrossRef] [PubMed]
Cai, T. , Liu, W. , and Xia, Y. , 2013, “ Two-Sample Covariance Matrix Testing and Support Recovery in High-Dimensional and Sparse Settings,” J. Am. Stat. Assoc., 108(501), pp. 265–277. [CrossRef]
DeGroot, M. H. , Schervish, M. J. , Fang, X. , Lu, L. , and Li, D. , 1986, Probability and Statistics, Vol. 2, Addison-Wesley, Reading, MA.
Hastie, T. , Tibshirani, R. , Friedman, J. , Hastie, T. , Friedman, J. , and Tibshirani, R. , 2009, The Elements of Statistical Learning, Vol. 2, Springer, New York. [CrossRef]
Kuhn, M. , and Johnson, K. , 2013, Applied Predictive Modeling, Springer, New York. [CrossRef]
Rasmussen, C. E. , and Williams, C. K. , 2006, Gaussian Processes for Machine Learning, Vol. 1, MIT Press, Cambridge, MA.
Seber, G. A. , and Lee, A. J. , 2012, Linear Regression Analysis, Vol. 936, Wiley, Hoboken, NJ.
Hamilton, J. D. , 1994, Time Series Analysis, Vol. 2, Princeton University Press, Princeton, NJ.
Engle, R. , and Kelly, B. , 2012, “ Dynamic Equicorrelation,” J. Bus. Econ. Stat., 30(2), pp. 212–228. [CrossRef]
Alexander, C. , and Lazar, E. , 2006, “ Normal Mixture Garch (1, 1): Applications to Exchange Rate Modelling,” J. Appl. Econ., 21(3), pp. 307–336. [CrossRef]
Székely, G. J. , and Rizzo, M. L. , 2013, “ Energy Statistics: A Class of Statistics Based on Distances,” J. Stat. Plann. Inference, 143(8), pp. 1249–1272. [CrossRef]
McMahan, J. A. , 2016, “ LinVer-Matlab,” GitHub, Inc., Beaverton, OR, accessed Dec. 7, 2016, https://github.com/jmcmahan/LinVer-Matlab


Grahic Jump Location
Fig. 1

Comparison of DRAM-computed densities with the analytically computed densities for case 3 (β̂,λ̂,ϕ̂ calibrated) with AR(1) correlated observation error and noninformative prior: (a) β̂1 correct code, (b) β̂1 incorrect code, (c) β̂2 correct code, and (d) β̂2 incorrect code

Grahic Jump Location
Fig. 2

Comparison of DRAM-computed densities with the analytically computed densities for case 3 (β̂,λ̂,ϕ̂ calibrated) with AR(1) correlated observation error and noninformative prior: (a) λ̂ correct code, (b) λ̂ incorrect code, (c) ϕ̂ correct code, and (d) ϕ̂ incorrect code

Grahic Jump Location
Fig. 3

Comparison of DRAM-computed densities with the analytically computed densities for case 3 (β̂,λ̂,ϕ̂ calibrated) with AR(1) correlated observation error and Gaussian prior: (a) β̂1 correct code, (b) β̂1 incorrect code, (c) β̂2 correct code, and (d) β̂2 incorrect code

Grahic Jump Location
Fig. 4

Comparison of DRAM-computed densities with the analytically computed densities for case 3 (β̂,λ̂,ϕ̂ calibrated) with AR(1) correlated observation error and Gaussian prior: (a) λ̂ correct code, (b) λ̂ incorrect code, (c) ϕ̂ correct code, and (d) ϕ̂ incorrect code




Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In