0
Guest Editorial

Introduction: The 2014 Sandia Verification and Validation Challenge Workshop OPEN ACCESS

[+] Author and Article Information
Kenneth T. Hu, Vicente Romero

Mem. ASME
V&V, UQ, and Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828

Brian Carnes

Mem. ASME
V&V, UQ, and Credibility Processes Department,
Sandia National Laboratories,
P.O. Box 5800, MS 0828,
Albuquerque, NM 87185-0828

The United States Government retains, and by accepting the article for publication, the publisher acknowledges that the United States Government retains, a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for United States government purposes.

J. Verif. Valid. Uncert 1(1), 010301 (Feb 19, 2016) (4 pages) Paper No: VVUQ-15-1052; doi: 10.1115/1.4032569 History: Received November 10, 2015; Revised January 05, 2016
FIGURES IN THIS ARTICLE

Over the past 20 years, Sandia National Laboratories (Sandia) and ASME [16] have played major roles in the development of V&V concepts, methods, and processes by building on the work of many other individuals and organizations [712]. Along the way, Sandia has organized workshops to demonstrate the state-of-the-art and highlight particular V&V issues. The most recent, the 2014 Sandia V&V Challenge Workshop, was held at the third ASME V&V Symposium in Las Vegas, NV, on May 8, 2014. As a challenge workshop, it was built around a challenge problem crafted to exercise specific ideas and spark discussions. The problem was released to the community prior to the workshop to allow participants time to prepare their responses. Challenge problems provide a common basis upon which a diverse group of participants can demonstrate methods and ideas and the workshops have become a benchmark of sorts for the V&V community. The 2014 workshop focused on the overall V&V strategies to address the challenge problem and also fostered discussion about how a V&V strategy can establish or assess the credibility of a specific prediction.

This special edition of the ASME Journal of Verification, Validation and Uncertainty Quantification documents the workshop and is organized as follows. This introductory paper discusses the history of workshops and how the goals have evolved with the V&V community—leading to the 2014 workshop. The next paper describes the problem statement and how the data provided for the problem were created (the “truth model”) [13]. The full problem description, posed to the participants in 2013, is available as a technical report [14] but does not include the truth model. These two papers are followed by five contributions that describe the participants' responses to the challenge problem [1519]. Two discussion papers are also included, in which the authors do not solve the problem but instead address relevant V&V topics while using the challenge problem as context [20,21]. The special edition concludes with a summary paper that describes the responses and lessons learned [22].

The 2002 Workshop on Alternative Representations of Epistemic Uncertainty dealt with uncertainty quantification (UQ), or how to characterize, model, and propagate uncertainty with both epistemic and aleatoric nature [23,24]. The workshop highlighted the importance of uncertainty, and the need to quantify uncertainty and understand its impact on model predictions. This workshop's goal was to compare different concepts and algorithms to describe predictions that rely on uncertain information. The participants found that each method had advantages and drawbacks, and they concluded that a healthy diversity of opinions would persist in the community [25].

The 2006 Validation Challenge Workshop posed problems with a hierarchical flow of information (i.e., materials level calibration, which informs validation at an “exploratory” level and “accreditation” level) [2630]. The hierarchy is a visualization of how data and model predictions are used during multiple V&V analyses [31]. It describes at a high level how the data and models relate to each other. This forms the basis for the selection and execution of V&V analyses as part of a larger strategy to improve predictions or assess credibility. The participants in the 2006 workshop had to apply multiple V&V methods across a predetermined hierarchy, to make a final prediction and satisfy a regulatory requirement. These challenge problems illustrated that (1) uncertainty is pervasive throughout modeling work and V&V, (2) quantitative methods are needed to assess the uncertainty across the hierarchy, and (3) a wide range of predictions and uncertainty estimates should be expected—even when the hierarchical information flow is specified [3235]. Several other UQ and V&V workshops have focused on methods and theory [36], including some that did not contain a challenge problem [11,12,37].

By 2012, the workshop organizers felt that adequate attention was being paid to research on V&V theory and methods, and adoption of V&V methods and principles was spreading rapidly. Many journals from diverse engineering disciplines were publishing V&V-focused papers and the topic was popular at conferences—notably the ASME V&V Symposium. Both academia and industry were researching and applying V&V methods, however, very little was known about how to utilize the results. Decision makers and customers were struggling to understand and interpret the evidence gathered from V&V methods. Unfortunately, the two ends of this spectrum (research versus applications) are not easily connected for several reasons: (1) the V&V community of practice is relatively small, (2) a huge range of expertise is required to perform all the V&V activities, (3) the incentives of researchers (novel methods and skills, broad and general impact, and publishing ideas) and practicing engineers (delivering results, creating a process tailored to one organization, and maintaining competitive advantage) have limited overlap, and (4) the benefits of V&V are hard to measure and communicate. The organizers saw the need to continue developing V&V methods, provide the community with V&V experience, and investigate the decision-making process. In response, they launched efforts to develop a new problem and hold a third challenge workshop.

The 2014 Sandia Verification and Validation Challenge Workshop was a daylong event with ten talks and follow-up discussions. After the organizers opened the workshop, nine participants presented responses to the challenge problem or their views on the role of V&V in decision support, followed by an open discussion. The participants were from academia, consulting firms, and the U.S. research laboratories, and were predominantly engineers and mathematicians. The workshop was open to all attendees of the ASME V&V Symposium and an estimated 100 people attended some or all of the talks and discussion.

Hu (Sandia National Laboratories) served as workshop moderator and introduced the challenge problem [38]. This was followed by six responses from: Shields et al. (John Hopkins University) [39,40], Roy et al. (Virginia Tech) [41], Chen et al. (Northwestern University) [42], Xi and Pan (University of Michigan—Dearborn) [43], Mahadevan and Mullins (Vanderbilt) [44], and Beghini and Hough (Sandia National Laboratories) [45]. For brevity, only the team leads are listed but each presentation was the work of a group. Three more speakers presented their thoughts on how V&V plays a role in decision making, Paez (consultant) gave an economic analysis of whether modeling, simulation, and V&V are worth the cost [46], Elele (U.S. Naval Air Warfare Center) discussed how V&V can be used to assess risk levels and impact on projects [47], and Brodrick (U.S. Naval Surface Warfare Center) discussed how models are currently used to support qualification activities, and a vision for how V&V and UQ should be incorporated into that process [48]. After the workshop, several of the participants continued to refine their solutions and discussion topics, which led to the creation of this special edition.

The scenario described in the challenge problem is an engineering project to predict the behavior of a population of simple storage tanks under a variety of loads. Various experimental studies have been completed, and limited data are available. In addition, a finite-element model has been developed to predict structural response—but not mechanical failure. The scenario ends in a decision whether the entire population should stay in service or needs to be replaced; however, the challenge is simplified into three tasks: predict probability of failure plus an uncertainty estimate, assess the credibility of the prediction, and describe the V&V strategy used. This paper reveals the motivations and the important features; the problem statement is described in the following papers [13,14].

The challenge problems must trade-off the flexibility to allow a variety of V&V approaches and the constraints required to focus the participants' attention. This problem de-emphasizes individual V&V methodologies and focuses on strategies to gather and integrate V&V evidence. This was accomplished by defining an open-ended set of objectives—in contrast to the more prescriptive nature of previous challenge problems [24,2830,36]. The principle challenge was to develop a defensible V&V strategy, which prioritized which analyses to complete and which methods to use. The participants were free to choose how to use the data and model to achieve the ultimate prediction and had the opportunity—but not the obligation—to perform many analyses: UQ, calibration, solution verification, validation, etc. In addition, participants had to integrate the results and communicate their conclusions. This parallels the projects tackled by engineering organizations; the purpose is to supply credible advice to support technical decisions and there is no set of instructions to follow. This focus on V&V strategy is a small step toward bridging the gap from theory and methods to the processes needed to support decisions. The question is now: Which V&V analyses are the appropriate analyses to perform, in order to make the desired predictions? The next question will be: How do these analyses support and impact the ultimate decision?

To successfully investigate these questions about V&V strategies, the challenge problem required a great deal of complexity and context. The scope of the problem enabled participants to construct their own hierarchy based on the available data and models. This also raised the question of how to aggregate or integrate uncertainty estimates from different analyses—for example, uncertainty from a mesh convergence study, uncertainty and variability of material properties, plus the effects of measurement error. This requires careful considerations of the “meaning” of each uncertainty and the implications for credibility, in addition to methods for UQ.

The large scope also required a large commitment from the participants, and the expectation was that all participants would be limited by time, resources, and expertise—it is a major challenge to assemble a team that includes experts in verification, validation, UQ, physics domains, and systems engineering. That being said, several compromises were made to ensure the problem could be solved in a reasonable timeframe.

  • The end decision of whether to replace the tanks is not part of the challenge. The problem lacks the required context, such as cost of the tanks, the state of the company, risk tolerances, etc. Instead, an arbitrary criterion is created for the probability of failure. As a consequence, participants focused on a specific prediction and did not have to speculate on the intended use of the model. The participants and audience were expected to be V&V analysts and not systems engineers or decision makers, so this challenge was limited to fit their backgrounds. Nevertheless, the additional context was included to raise awareness of how V&V eventually impacts decisions.

  • The data, model, and code were supplied. This removed all choices about model development and design of experiments, but greatly reduced the workload. Four finite-element mesh resolutions were available, based on the same geometry, which could be solved using the provided code. The code was described as a finite-element solver, but was actually an algebraic function. This enabled anyone to easily and cheaply run the code, but the model behavior was not the same as a finite-element model—and code verification was not sensible. The data were created from a truth model—not physical experiments.

  • Only a single structural mechanics example was provided. The organizers wanted a compelling engineering problem rather than a generic mathematical exercise, but needed to keep it simple and accessible for engineers from any physics discipline background. Inevitably, some domain knowledge was helpful in interpreting the results.

The challenge workshop was not a competition and there is no “best” strategy. The responses were not scored or ranked. In fact, the open-ended problem formulation meant that direct comparison of responses was impossible. Discrepancies could not be traced to a single cause because so many different assumptions and method choices were made by each team. Also, no limits were placed on the time invested in the problem, and the participating teams had a wide range of effort levels and priorities. The only step taken to ensure fairness was to provide all the teams access to the same information and instructions. The organizers did not collaborate with any of the participants or reveal the source of the data prior to this special edition. The sources of the data, the truth model, are now available [13]. Again, the organizers stress that scoring the approaches by comparing to the “truth” is not the intention of this activity—indeed, it is not clearly understood how to objectively judge any of the approaches at all. The organizers are not interested in the “correct result”; the goal is to learn about different V&V strategies.

Sandia has used challenge workshops as benchmarks of the V&V community. The community has evolved from developing individual UQ and V&V methods, for investigating how to integrate analyses across a hierarchy and now designing the appropriate hierarchy. Development and demonstration of individual methods are still absolutely critical, but this workshop emphasized V&V strategy as the current challenge. The quality of the workshop responses and the discussions at the ASME V&V Symposium indicate that the V&V community is growing, and V&V theory and methods are maturing. Again, the workshop saw a healthy diversity of V&V strategies, which reflect the different priorities and backgrounds of the participants. A summary of the responses and commentary is included in the following paper [22]. The major lesson is that we need more experience with a larger variety of case studies and we need to learn how to judge the efficacies of V&V strategies.

Looking forward, a major obstacle will be the vast scope of V&V, which relates to the fields of statistics, probability theory, numerical algorithms, systems engineering, and many intermediate topics. It is difficult to merge quantitative and rigorous methods with the qualitative and subjective aspects of credibility. The next challenge may concern the role of V&V with respect to modeling and simulation and the decision-making process.

The workshop organizers hope that the V&V community will build upon the work in this special edition and make use of this challenge problem (and those from prior workshops). These problems are excellent case studies for teaching or demonstrating V&V approaches and concepts and will spark discussions about the role of V&V in engineering work.

This challenge problem and the resulting workshop were made possible with support from the Sandia National Laboratories and ASME. We wish to thank Greg Weirs, Laura Swiler, Walt Witkowski, and the Dakota team at the Sandia National Laboratories, and Ryan Crane and the V&V Standards committees at ASME for helping make the workshop a success. We are especially grateful to the workshop participants, who made this special edition possible, including those whose work does not appear in this issue: Michael Shields, Thomas Brodrick, and James Elele. Sandia National Laboratories is a multiprogram laboratory managed and operated by the Sandia Corporation, a wholly owned subsidiary of the Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under Contract No. DE-AC04-94AL85000.

Oberkampf, W. L. , Pilch, M. , and Trucano, T. G. , 2007, “ Predictive Capability Maturity Model for Computational Modeling and Simulation,” Sandia National Laboratories, Report No. SAND2007-5948.
Trucano, T. G. , Pilch, M. , and Oberkampf, W. L. , 2002, “ General Concepts for Experimental Validation of ASCI Code Applications,” Sandia National Laboratories, Report No. SAND2002-0341.
Trucano, T. G. , Pilch, M. , and Oberkampf, W. L. , 2003, “ On the Role of Code Comparisons in Verification and Validation,” Sandia National Laboratories, Report No. SAND2003-2752.
ASME V&V 20, 2009, Standard for Verification and Validation in Computational Fluids and Heat Transfer, The American Society of Mechanical Engineers, New York.
ASME V&V 10-2006, 2006, Guide for Verification and Validation in Computational Solid Mechanics, The American Society of Mechanical Engineers, New York.
Oberkampf, W. L. , and Barone, M. F. , 2006, “ Measures of Agreement Between Computation and Experiment: Validation Metrics,” J. Comput. Phys., 217(1), pp. 5–36. [CrossRef]
Oberkampf, W. L. , 1998, “ Bibliography for Verification and Validation in Computational Simulation,” Sandia National Laboratories, Report No. SAND98-2041.
Schwer, L. E. , and Oberkampf, W. L. , 2007, “ Special Issue on Verification and Validation,” Eng. Comput., 23(4), pp. 243–244. [CrossRef]
Oberkampf, W. L. , Sindir, M. , and Conlisk, A. T. , 1998, “ Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,” AIAA Paper No. G-077-1998.
Nitta, C. K. , and Logan, R. W. , 2004, “ ASCI V&V at LLNL: An Unclassified Bibliography,” Lawrence Livermore National Laboratory, Report No. UCRL-AR-203864.
Pace, D. , 2002, “ Foundations'02 Overview,” Foundations'02 a Workshop on Model and Simulation Verification and Validation for the 21st Century, D. Pace , ed., JHU/APL, Laurel, MD.
Youngblood, S. M. , 2004, “ Roadmap for VV&A Technology Advancement,” Foundations'04: A Workshop for V&V in the 21st Century, Arizona State University, Phoenix, AZ, Defense Modeling & Simulation Office.
Hu, K. T. , and Orient, G. E. , 2016, “ The 2014 Sandia V&V Challenge: Problem Statement,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011001. [CrossRef]
Hu, K. T. , 2013, “ 2014 V&V Challenge: Problem Statement,” Sandia National Laboratories, Report No. SAND2013-10486P.
Choudhary, A. , Voyles, I. T. , Roy, C. J. , Oberkampf, W. L. , and Patil, M. , 2016, “ Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011003. [CrossRef]
Li, W. , Chen, S. , Jiang, Z. , Apley, D. W. , Lu, Z. , and Chen, W. , 2016, “ Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011004. [CrossRef]
Mullins, J. , and Mahadevan, S. , 2016, “ Bayesian Information Fusion for Model Calibration, Validation, and Prediction,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011006. [CrossRef]
Beghini, L. L. , and Hough, P. D. , 2016, “ Sandia V&V Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011002. [CrossRef]
Xi, Z. , and Yang, R. J. , 2016, “ Reliability Analysis With Model Uncertainty Coupling With Parameter and Experimental Uncertainties: A Case Study of 2014 V&V Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011005. [CrossRef]
Hu, K. T. , and Paez, T. , 2016, “ Why Do Verification and Validation?” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011008. [CrossRef]
Paez, P. J. , Paez, T. , and Hasselman, T. K. , 2016, “ Economic Analysis of Model Validation for a Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011007. [CrossRef]
Schroeder, B. B. , Hu, K. T. , Mullins, J. G. , and Winokur, J. G. , 2016, “ Summary of the 2014 Sandia V&V Challenge Workshop,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 015501. [CrossRef]
Helton, J. C. , and Oberkampf, W. L. , 2004, “ Alternative Representations of Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 1–10. [CrossRef]
Oberkampf, W. L. , Helton, J. C. , Joslyn, C. A. , Wojtkiewicz, S. F. , and Ferson, S. , 2004, “ Challenge Problems: Uncertainty in System Response Given Uncertain Parameters,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 11–19. [CrossRef]
Ferson, S. , Joslyn, C. A. , Helton, J. C. , Oberkampf, W. L. , and Sentz, K. , 2004, “ Summary From the Epistemic Uncertainty Workshop: Consensus Amid Diversity,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 355–369. [CrossRef]
Pilch, M. , 2008, “ Preface,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2373–2374. [CrossRef]
Hills, R. G. , Pilch, M. , Dowding, K. J. , Red-Horse, J. , Paez, T. L. , Babuška, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2375–2380. [CrossRef]
Babuska, I. , Nobile, F. , and Tempone, R. , 2008, “ Formulation of the Static Frame Problem,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2496–2499. [CrossRef]
Dowding, K. J. , Pilch, M. , and Hills, R. G. , 2008, “ Formulation of the Thermal Problem,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2385–2389. [CrossRef]
Red-Horse, J. R. , and Paez, T. L. , 2008, “ Sandia National Laboratories Validation Workshop: Structural Dynamics Application,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2578–2584. [CrossRef]
Oberkampf, W. L. , and Trucano, T. G. , 2002, “ Verification and Validation in omputational Fluid Dynamics,” Prog. Aerosp. Sci., 38(3), pp. 209–272. [CrossRef]
Dowding, K. J. , Red-Horse, J. , Hills, R. G. , Paez, T. L. , Babuska, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2381–2384. [CrossRef]
Paez, T. L. , and Red-Horse, J. R. , 2008, “ Structural Dynamics Challenge Problem: Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2660–2665. [CrossRef]
Hills, R. G. , Dowding, K. J. , and Swiler, L. , 2008, “ Thermal Challenge Problem: Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2490–2495. [CrossRef]
Babuska, I. , and Tempone, R. , 2008, “ Static Frame Challenge Problem: Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2572–2577. [CrossRef]
Luis, G. C. , Sean, P. K. , and Daniel, P. G. , 2014, “ The NASA Langley Multidisciplinary Uncertainty Quantification Challenge,” 16th AIAA Non-Deterministic Approaches Conference, American Institute of Aeronautics and Astronautics.
Webster, C. , Collis, S. , Gunzburger, M. , Trucano, T. , and Womble, D. , 2015, “ Sandia CSRI Workshop on Mathematical Methods for Validation and Verification,” Last accessed Oct. 26, 2015, http://www.cs.sandia.gov/CSRI/Workshops/2007/MMVV/index.html
Hu, K. T. , 2014, “ The Sandia National Laboratories 2014 Verification & Validation Challenge Workshop,” ASME Paper No. V&V2014-7211.
Shields, M. D. , Teferra, K. , and Kim, H. , 2014, “ V&V Challenge Problem: An Efficient Monte Carlo Method Incorporating the Effects of Model Error,” ASME Paper No. V&V2014-7214.
Shields, M. D. , Teferra, K. , Hapij, A. , and Daddazio, R. P. , 2015, “ Refined Stratified Sampling for Efficient Monte Carlo Based Uncertainty Quantification,” Reliab. Eng. Syst. Saf., 142, pp. 310–325. [CrossRef]
Roy, C. , Choudhary, A. , Voyles, I. T. , and Patil, M. , 2014, “ Probability Bounds Analysis Applied to the Sandia V&V Challenge Problem,” ASME Paper No. V&V2014-7219.
Chen, W. , Apley, D. , Li, W. , Jiang, Z. , and Chen, S. , 2014, “ A Bayesian Inference Approach to the Validation Challenge Problem,” ASME Paper No. V&V2014-7216.
Xi, Z. , and Pan, H. , 2014, “ Copula-Based Validation Approach for 2014 V&V Challenge Problem,” ASME Paper No. V&V2014-7220.
Mahadevan, S. , and Mullins, J. , 2014, “ Bayesian Information Fusion for Model Validation and Prediction,” ASME Paper No. V&V2014-7213.
Beghini, L. L. , and Hough, P. , 2014, “ Surrogate-Based V&V,” ASME Paper No. V&V2014-7226.
Paez, P. J. , Paez, T. , and Hasselman, T. K. , 2014, “ The Economics of Model Validation,” ASME Paper No. V&V2014-7215.
Elele, J. , and Hall, D. , 2014, “ Top-Down Risk Based Approach to M&S VV&A,” ASME Paper No. V&V2014-7210.
Brodrick, T. , 2014, “ Sample Problem From a DOD Perspective,” ASME Paper No. V&V2014-7217.
Copyright © 2016 by ASME
View article in PDF format.

References

Oberkampf, W. L. , Pilch, M. , and Trucano, T. G. , 2007, “ Predictive Capability Maturity Model for Computational Modeling and Simulation,” Sandia National Laboratories, Report No. SAND2007-5948.
Trucano, T. G. , Pilch, M. , and Oberkampf, W. L. , 2002, “ General Concepts for Experimental Validation of ASCI Code Applications,” Sandia National Laboratories, Report No. SAND2002-0341.
Trucano, T. G. , Pilch, M. , and Oberkampf, W. L. , 2003, “ On the Role of Code Comparisons in Verification and Validation,” Sandia National Laboratories, Report No. SAND2003-2752.
ASME V&V 20, 2009, Standard for Verification and Validation in Computational Fluids and Heat Transfer, The American Society of Mechanical Engineers, New York.
ASME V&V 10-2006, 2006, Guide for Verification and Validation in Computational Solid Mechanics, The American Society of Mechanical Engineers, New York.
Oberkampf, W. L. , and Barone, M. F. , 2006, “ Measures of Agreement Between Computation and Experiment: Validation Metrics,” J. Comput. Phys., 217(1), pp. 5–36. [CrossRef]
Oberkampf, W. L. , 1998, “ Bibliography for Verification and Validation in Computational Simulation,” Sandia National Laboratories, Report No. SAND98-2041.
Schwer, L. E. , and Oberkampf, W. L. , 2007, “ Special Issue on Verification and Validation,” Eng. Comput., 23(4), pp. 243–244. [CrossRef]
Oberkampf, W. L. , Sindir, M. , and Conlisk, A. T. , 1998, “ Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,” AIAA Paper No. G-077-1998.
Nitta, C. K. , and Logan, R. W. , 2004, “ ASCI V&V at LLNL: An Unclassified Bibliography,” Lawrence Livermore National Laboratory, Report No. UCRL-AR-203864.
Pace, D. , 2002, “ Foundations'02 Overview,” Foundations'02 a Workshop on Model and Simulation Verification and Validation for the 21st Century, D. Pace , ed., JHU/APL, Laurel, MD.
Youngblood, S. M. , 2004, “ Roadmap for VV&A Technology Advancement,” Foundations'04: A Workshop for V&V in the 21st Century, Arizona State University, Phoenix, AZ, Defense Modeling & Simulation Office.
Hu, K. T. , and Orient, G. E. , 2016, “ The 2014 Sandia V&V Challenge: Problem Statement,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011001. [CrossRef]
Hu, K. T. , 2013, “ 2014 V&V Challenge: Problem Statement,” Sandia National Laboratories, Report No. SAND2013-10486P.
Choudhary, A. , Voyles, I. T. , Roy, C. J. , Oberkampf, W. L. , and Patil, M. , 2016, “ Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011003. [CrossRef]
Li, W. , Chen, S. , Jiang, Z. , Apley, D. W. , Lu, Z. , and Chen, W. , 2016, “ Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011004. [CrossRef]
Mullins, J. , and Mahadevan, S. , 2016, “ Bayesian Information Fusion for Model Calibration, Validation, and Prediction,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011006. [CrossRef]
Beghini, L. L. , and Hough, P. D. , 2016, “ Sandia V&V Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011002. [CrossRef]
Xi, Z. , and Yang, R. J. , 2016, “ Reliability Analysis With Model Uncertainty Coupling With Parameter and Experimental Uncertainties: A Case Study of 2014 V&V Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011005. [CrossRef]
Hu, K. T. , and Paez, T. , 2016, “ Why Do Verification and Validation?” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011008. [CrossRef]
Paez, P. J. , Paez, T. , and Hasselman, T. K. , 2016, “ Economic Analysis of Model Validation for a Challenge Problem,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 011007. [CrossRef]
Schroeder, B. B. , Hu, K. T. , Mullins, J. G. , and Winokur, J. G. , 2016, “ Summary of the 2014 Sandia V&V Challenge Workshop,” ASME J. Verif., Validation, Uncertainty Quantif., 1(1), p. 015501. [CrossRef]
Helton, J. C. , and Oberkampf, W. L. , 2004, “ Alternative Representations of Epistemic Uncertainty,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 1–10. [CrossRef]
Oberkampf, W. L. , Helton, J. C. , Joslyn, C. A. , Wojtkiewicz, S. F. , and Ferson, S. , 2004, “ Challenge Problems: Uncertainty in System Response Given Uncertain Parameters,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 11–19. [CrossRef]
Ferson, S. , Joslyn, C. A. , Helton, J. C. , Oberkampf, W. L. , and Sentz, K. , 2004, “ Summary From the Epistemic Uncertainty Workshop: Consensus Amid Diversity,” Reliab. Eng. Syst. Saf., 85(1–3), pp. 355–369. [CrossRef]
Pilch, M. , 2008, “ Preface,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2373–2374. [CrossRef]
Hills, R. G. , Pilch, M. , Dowding, K. J. , Red-Horse, J. , Paez, T. L. , Babuška, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2375–2380. [CrossRef]
Babuska, I. , Nobile, F. , and Tempone, R. , 2008, “ Formulation of the Static Frame Problem,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2496–2499. [CrossRef]
Dowding, K. J. , Pilch, M. , and Hills, R. G. , 2008, “ Formulation of the Thermal Problem,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2385–2389. [CrossRef]
Red-Horse, J. R. , and Paez, T. L. , 2008, “ Sandia National Laboratories Validation Workshop: Structural Dynamics Application,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2578–2584. [CrossRef]
Oberkampf, W. L. , and Trucano, T. G. , 2002, “ Verification and Validation in omputational Fluid Dynamics,” Prog. Aerosp. Sci., 38(3), pp. 209–272. [CrossRef]
Dowding, K. J. , Red-Horse, J. , Hills, R. G. , Paez, T. L. , Babuska, I. , and Tempone, R. , 2008, “ Validation Challenge Workshop Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2381–2384. [CrossRef]
Paez, T. L. , and Red-Horse, J. R. , 2008, “ Structural Dynamics Challenge Problem: Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2660–2665. [CrossRef]
Hills, R. G. , Dowding, K. J. , and Swiler, L. , 2008, “ Thermal Challenge Problem: Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2490–2495. [CrossRef]
Babuska, I. , and Tempone, R. , 2008, “ Static Frame Challenge Problem: Summary,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2572–2577. [CrossRef]
Luis, G. C. , Sean, P. K. , and Daniel, P. G. , 2014, “ The NASA Langley Multidisciplinary Uncertainty Quantification Challenge,” 16th AIAA Non-Deterministic Approaches Conference, American Institute of Aeronautics and Astronautics.
Webster, C. , Collis, S. , Gunzburger, M. , Trucano, T. , and Womble, D. , 2015, “ Sandia CSRI Workshop on Mathematical Methods for Validation and Verification,” Last accessed Oct. 26, 2015, http://www.cs.sandia.gov/CSRI/Workshops/2007/MMVV/index.html
Hu, K. T. , 2014, “ The Sandia National Laboratories 2014 Verification & Validation Challenge Workshop,” ASME Paper No. V&V2014-7211.
Shields, M. D. , Teferra, K. , and Kim, H. , 2014, “ V&V Challenge Problem: An Efficient Monte Carlo Method Incorporating the Effects of Model Error,” ASME Paper No. V&V2014-7214.
Shields, M. D. , Teferra, K. , Hapij, A. , and Daddazio, R. P. , 2015, “ Refined Stratified Sampling for Efficient Monte Carlo Based Uncertainty Quantification,” Reliab. Eng. Syst. Saf., 142, pp. 310–325. [CrossRef]
Roy, C. , Choudhary, A. , Voyles, I. T. , and Patil, M. , 2014, “ Probability Bounds Analysis Applied to the Sandia V&V Challenge Problem,” ASME Paper No. V&V2014-7219.
Chen, W. , Apley, D. , Li, W. , Jiang, Z. , and Chen, S. , 2014, “ A Bayesian Inference Approach to the Validation Challenge Problem,” ASME Paper No. V&V2014-7216.
Xi, Z. , and Pan, H. , 2014, “ Copula-Based Validation Approach for 2014 V&V Challenge Problem,” ASME Paper No. V&V2014-7220.
Mahadevan, S. , and Mullins, J. , 2014, “ Bayesian Information Fusion for Model Validation and Prediction,” ASME Paper No. V&V2014-7213.
Beghini, L. L. , and Hough, P. , 2014, “ Surrogate-Based V&V,” ASME Paper No. V&V2014-7226.
Paez, P. J. , Paez, T. , and Hasselman, T. K. , 2014, “ The Economics of Model Validation,” ASME Paper No. V&V2014-7215.
Elele, J. , and Hall, D. , 2014, “ Top-Down Risk Based Approach to M&S VV&A,” ASME Paper No. V&V2014-7210.
Brodrick, T. , 2014, “ Sample Problem From a DOD Perspective,” ASME Paper No. V&V2014-7217.

Figures

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In