0
Discussion

Summary of the 2014 Sandia Verification and Validation Challenge Workshop

[+] Author and Article Information
Benjamin B. Schroeder

V&V, UQ, Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828
e-mail: bbschro@sandia.gov

Kenneth T. Hu

V&V, UQ, Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828
e-mail: khu@sandia.gov

Joshua G. Mullins

V&V, UQ, Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828
e-mail: jmullin@sandia.gov

Justin G. Winokur

V&V, UQ, Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828
e-mail: jgwinok@sandia.gov

Manuscript received December 4, 2015; final manuscript received January 13, 2016; published online February 19, 2016. Editor: Ashley F. Emery.

J. Verif. Valid. Uncert 1(1), 015501 (Feb 19, 2016) (9 pages) Paper No: VVUQ-15-1055; doi: 10.1115/1.4032563 History: Received December 04, 2015; Revised January 13, 2016

A discussion of the five responses to the 2014 Sandia Verification and Validation (V&V) Challenge Problem, presented within this special issue, is provided hereafter. Overviews of the challenge problem workshop, workshop participants, and the problem statement are also included. Brief summations of teams' responses to the challenge problem are provided. Issues that arose throughout the responses that are deemed applicable to the general verification, validation, and uncertainty quantification (VVUQ) community are the main focal point of this paper. The discussion is oriented and organized into big picture comparison of data and model usage, VVUQ activities, and differentiating conceptual themes behind the teams' VVUQ strategies. Significant differences are noted in the teams' approaches toward all VVUQ activities, and those deemed most relevant are discussed. Beyond the specific details of VVUQ implementations, thematic concepts are found to create differences among the approaches; some of the major themes are discussed. Finally, an encapsulation of the key contributions, the lessons learned, and advice for the future are presented.

FIGURES IN THIS ARTICLE
<>
Copyright © 2016 by ASME
Your Session has timed out. Please sign back in to continue.

References

Hu, K. T. , Carnes, B. , and Romero, V. , “ The 2014 Sandia V&V Challenge Problem Workshop,” J. Verif., Validation Uncertainty Quantif., 1(1).
Hu, K. T. , 2013, “ 2014 V&V Challenge: Problem Statement,” Sandia National Laboratories, Albuquerque, NM, Technical Report No. SAND2013-10486P.
Hu, K. T. , and Orient, G. E. , “ The 2014 Sandia V&V Challenge Problem Statement,” J. Verif., Validation Uncertainty Quantif., 1(1).
Helton, J. C. , and Oberkampf, W. L. , 2004, “ Alternative Representations of Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 85, pp. 1–10. [CrossRef]
Hills, R. G. , Pilch, M. , Dowding, K. J. , Red-Horse, J. , Paez, T. L. , Babuška, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop,” Comput. Methods Appl. Mech., 197, pp. 2375–2380. [CrossRef]
Beghini, L. L. , and Hough, P. D. , “ Sandia V&V Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility,” J. Verif., Validation Uncertainty Quantif., 1(1).
Choudhary, A. , Voyles, I . T. , Roy, C. J. , Oberkampf, W. L. , and Patil, M. , “ Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem,” J. Verif., Validation Uncertainty Quantif., 1(1).
Li, W. , Chen, S. , Jiang, Z. , Apley, D. W. , Lu, Z. , and Chen, W. , “ Integrating Calibration, Bias Correction, and Machine Learning for the Challenge Problem,” J. Verif., Validation Uncertainty Quantif., 1(1).
Xi, Z. , and Yang, R.-J. , “ Reliability Analysis With Model Uncertainty Coupling With Parameter and Experimental Uncertainties: A Case Study of 2014 V&V Challenge Problem,” J. Verif., Validation Uncertainty Quantif., 1(1).
Mullins, J. , and Mahadevan, S. , “ Bayesian Information Fusion for Model Calibration, Validation, and Prediction,” J. Verif., Validation Uncertainty Quantif., 1(1).
Hu, K. T. , and Paez, T. L. , “ Why Do Verification and Validation?” J. Verif., Validation Uncertainty Quantif., 1(1).
Paez, P. J. , Paez, T. L. , and Hasselman, T. J. , “ The Economics of V&V,” J. Verif., Validation Uncertainty Quantif., 1(1).
Shields, M. D. , Teferra, K. , and Kim, H. , 2014. “ V&V Challenge Problem: An Efficient Monte Carlo Method Incorporating the Effects of Model Error,” ASME Paper No. V&V2014-7214.
Adams, B. M. , Bauman, L. E. , Bohnhoff, W. J. , Dalbey, K. R. , Eddy, J. P. , Ebieda, M. S. , Eldred, M. S. , Hough, P. D. , Hu, K. T. , Jakeman, J. D. , Swiler, L. P. , Stephens, J. A. , Vigil, D. M. , and Wildey, T. M. , 2014, “ Dakota—A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.1 User's Manual,” Sandia National Laboratories, Albuquerque, NM, Technical Report No. SAND2014-4633.
Bichon, B. J. , Eldred, M. S. , Swiler, L. P. , Mahadevan, S. , and McFarland, J. M. , 2008, “ Efficient Global Reliability Analysis for Nonlinear Implicit Performance Functions,” AIAA J., 46(10), pp. 2459–2468. [CrossRef]
Oberkampf, W. , Pilch, M. , and Trucano, T. , 2007, “ Predictive Capability Maturity Model for Computational Modeling and Simulation,” Sandia National Laboratories, Albuquerque, NM, Technical Report No. SAND2007-5948.
Ferson, S. , Oberkampf, W. L. , and Ginzburg, L. , 2008, “ Model Validation and Predictive Capability for the Thermal Challenge Problem,” Comput. Methods Appl. Mech., 197, pp. 2408–2430. [CrossRef]
Voyles, I. T. , and Roy, C. J. , 2014, “ Evaluation of Model Validation Techniques in the Presence of Uncertainty,” AIAA Paper No. 2014-0120.
Voyles, I . T. , and Roy, C. J. , 2015, “ Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties,” AIAA Paper No. 2015-1374.
Kennedy, M. C. , and O'Hagan, A. , 2001, “ Bayesian Calibration of Computer Models,” J. R. Stat. Soc. B, 63(3), pp. 425–464. [CrossRef]
Oberkampf, W. L. , and Trucano, T. G. , 2002, “ Verification and Validation in Computational Fluid Dynamics,” Prog. Aerosp. Sci., 38(3), pp. 209–272. [CrossRef]
Ferson, S. , Joslyn, C. A. , Helton, J. C. , Oberkampf, W. L. , and Sentz, K. , 2004, “ Summary From the Epistemic Uncertainty Workshop: Consensus Amid Diversity,” Reliab. Eng. Syst. Saf., 85, pp. 355–369. [CrossRef]
Knupp, P. , and Salari, K. , 2002, Verification of Computer Codes in Computational Science and Engineering, CRC Press, Boca Raton, FL.
Roy, C. J. , 2005, “ Review of Code and Solution Verification Procedures for Computational Simulation,” J. Comput. Phys., 205(1), pp. 131–156. [CrossRef]
Saltelli, A. , Tarantola, S. , Campolongo, F. , and Ratto, M. , 2004, Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models, Wiley, Chichester, UK.
Trucano, T. G. , Swiler, L. P. , Igusa, T. , Oberkampf, W. L. , and Pilch, M. , 2006, “ Calibration, Validation, and Sensitivity Analysis: What's What,” Reliab. Eng. Syst. Saf., 91, pp. 1331–1357. [CrossRef]
ASME V&V 10 Committee, 2006, “ Guide for Verification and Validation in Computational Solid Mechanics,” The American Society of Mechanical Engineers, New York, Technical Report No. V&V 10-2006.
ASME V&V 20 Committee, 2009, “ Standard for Verification and Validation in Computational Fluids and Heat Transfer,” The American Society of Mechanical Engineers, New York, Technical Report No V&V 20-2009.
Ling, Y. , Mullins, J. , and Mahadevan, S. , 2014, “ Selection of Model Discrepancy Priors in Bayesian Calibration,” J. Comput. Phys., 276, pp. 665–680. [CrossRef]
AIAA Standards, 2002, “ Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,” AIAA Paper No. G-077-1998.
Hu, K. T. , 2014, “ The Sandia National Laboratories 2014 Verification & Validation Challenge Workshop,” ASME Paper No. V&V2014-7211.
Sargent, R. G. , 2011, “ Verification and Validation of Simulation Models,” Proceedings of the Winter Simulation Conference, Phoenix, AZ, pp.183–198.

Figures

Grahic Jump Location
Fig. 1

A VVUQ hierarchy for the challenge problem. Numbers identify the datasets, and the labels describe the system and environment of the experiments. The capabilities of the provided model are also indicated. (Reproduced with permission from Hu [2] (Figs. 2 and 3). Copyright 2013 by Sandia National Laboratories.)

Tables

Errata

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In