0
Research Papers

Statistical Approach for Computational Fluid Dynamics State-of-the-Art Assessment: N-Version Verification and Validation

[+] Author and Article Information
Frederick Stern

IIHR—Hydroscience and Engineering,
The University of Iowa,
Iowa City, IA 52242
e-mail: frederick-stern@uiowa.edu

Matteo Diez

IIHR—Hydroscience and Engineering,
The University of Iowa,
Iowa City, IA 52242;
CNR-INSEAN,
National Research Council—Marine
Technology Research Institute,
Rome 00128, Italy

Hamid Sadat-Hosseini, Hyunse Yoon

IIHR—Hydroscience and Engineering,
The University of Iowa,
Iowa City, IA 52242

Frans Quadvlieg

MARIN,
Maritime Research Institute Netherlands,
Wageningen 6708 PM, The Netherlands

1Corresponding author.

2Present address: Department of Mechanical and Energy Engineering, University of North Texas, Denton, TX 76207.

Manuscript received March 17, 2017; final manuscript received October 13, 2017; published online November 16, 2017. Assoc. Editor: Luis Eca.

J. Verif. Valid. Uncert 2(3), 031004 (Nov 16, 2017) (15 pages) Paper No: VVUQ-17-1014; doi: 10.1115/1.4038255 History: Received March 17, 2017; Revised October 13, 2017

A statistical approach for computational fluid dynamics (CFD) state-of-the-art (SoA) assessment is presented for specified benchmark test cases and validation variables, based on the combination of solution and N-version verification and validation (V&V). Solution V&V estimates the systematic numerical and modeling errors/uncertainties. N-version verification estimates the random simulation uncertainty. N-version validation estimates the random absolute error uncertainty, which is combined with the experimental and systematic numerical uncertainties into the SoA uncertainties and then used to determine whether or not the individual codes/simulations and the mean code are N-version validated at the USoAi and USoA intervals, respectively. The scatter is due to differences in models and numerical methods, grid types, domains, boundary conditions, and other setup parameters. Differences between codes/simulations and implementations are due to myriad possibilities for modeling and numerical methods and their implementation as CFD codes and simulation applications. Industrial CFD codes are complex software with many user options such that even in solving the same application, different results may be obtained by different users, not necessarily due to user error but rather the variability arising from the selection of various models, numerical methods, and setup options. Examples are shown for ship hydrodynamics applications using results from the Seventh CFD Ship Hydrodynamics and Second Ship Maneuvering Prediction Workshops. The role and relationship of individual code solution V&V versus N-version V&V and SoA assessment are discussed.

FIGURES IN THIS ARTICLE
<>
Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.

References

Coleman, H. , and Steele, G. , 1999, Experimentation and Uncertainty Analysis for Engineers, 2nd ed., Wiley, New York.
Stern, F. , Olivieri, A. , Shao, J. , Longo, J. , and Ratcliffe, T. , 2005, “ Statistical Approach for Estimating Intervals of Certification or Biases of Facilities or Measurement Systems Including Uncertainties,” ASME J. Fluids Eng., 127(2), pp. 604–610. [CrossRef]
Roache, P. J. , 1998, Verification and Validation in Computational Science and Engineering, Hermosa Publishers, Albuquerque, NM.
AIAA, 1998, “Guide for the Verification and Validation of Computational Fluid Dynamics Simulations,” AIAA Paper No. G-077-1998.
Oberkampf, W. , Suchyta, C. , and Benek, J. , 2016, “Validation Metrics for CFD,” AVT-246 Specialists' Meeting on Progress and Challenges in Validation Testing for Computational Fluid Dynamics, Avila, Spain, Sept. 26–28, Paper No. STO-MP-AVT-246.
Coleman, H. W. , and Stern, F. , 1997, “ Uncertainties in CFD Code Validation,” ASME J. Fluids Eng., 119(4), pp. 795–803. [CrossRef]
Roache, P. , 1998, “ Discussion: ‘Uncertainties and CFD Code Validation’ (Coleman, H. W., and Stern, F., 1997, ASME J. Fluids Eng., 119, pp. 795–803),” ASME J. Fluids Eng., 120(3), p. 635. [CrossRef]
Coleman, H. , and Stern, F. , 1998, “ Closure to ‘Discussion of “Uncertainties and CFD Code Validation”’ (1998, ASME J. Fluids Eng., 120, p. 635),” ASME J. Fluids Eng., 120(3), pp. 635–636. [CrossRef]
Stern, F. , Wilson, R. V. , Coleman, H. W. , and Paterson, E. G. , 2001, “ Comprehensive Approach to Verification and Validation of CFD Simulations—Part 1: Methodology and Procedures,” ASME J. Fluids Eng., 123(4), pp. 793–802. [CrossRef]
Wilson, R. V. , Stern, F. , Coleman, H. , and Paterson, E. , 2001, “ Comprehensive Approach to Verification and Validation of CFD Simulations–Part 2: Application for RANS Simulation of a Cargo/Container Ship,” ASME J. Fluids Eng., 123(4), pp. 803–810. [CrossRef]
Oberkampf, W. , 2002, “ Discussion: ‘Comprehensive Approach to Verification and Validation of CFD Simulations—Part 1: Methodology and Procedures’ (Stern, F., Wilson, R. V., Coleman, H. W., and Paterson, E. G., 2001, ASME J. Fluids Eng., 123, pp. 793–802),” ASME J. Fluids Eng., 124(3), pp. 809–810. [CrossRef]
Coleman, H. , 2002, “ Closure to “Discussion of ‘Comprehensive Approach to Verification and Validation of CFD Simulations—Part 1: Methodology and Procedures”’ (2002, ASME J. Fluids Eng., 124, p. 809),” ASME J. Fluids Eng., 124(3), p. 810. [CrossRef]
Stern, F. , and Wilson, R. , 2002, “ Closure to ‘Discussion of ‘Comprehensive Approach to Verification and Validation of CFD Simulations—Part 1: Methodology and Procedures’’ (2002, ASME J. Fluids Eng., 124, p. 809),” ASME J. Fluids Eng., 124(3), pp. 810–811. [CrossRef]
Roache, P. J. , 2003, “ Criticisms of the ‘Correction Factor’ Verification Method,” ASME J. Fluids Eng., 125(4), pp. 732–733. [CrossRef]
Wilson, R. , Shao, J. , and Stern, F. , 2004, “ Discussion: ‘Criticisms of the ‘Correction Factor’ Verification Method’ (Roache, P., 2003, ASME J. Fluids Eng., 125, pp. 732–733),” ASME J. Fluids Eng., 126(4), pp. 704–706. [CrossRef]
ASME, 2009, “Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer,” American Society of Mechanical Engineers, New York, Standard No. V&V 20-2009.
Xing, T. , and Stern, F. , 2010, “ Factors of Safety for Richardson Extrapolation,” ASME J. Fluids Eng., 132(6), p. 061403. [CrossRef]
Roache, P. J. , 2011, “ Discussion: ‘Factors of Safety for Richardson Extrapolation’ (Xing, T., and Stern, F., 2010, ASME J. Fluids Eng., 132, p. 061403),” ASME J. Fluids Eng., 133(11), p. 115501.
Xing, T. , and Stern, F. , 2011, “ Comments on Discussion of ‘Factors of Safety for Richardson Extrapolation,’” ASME J. Fluids Eng., 133(11), p. 115502. [CrossRef]
Eça, L. , and Hoekstra, M. , 2014, “ A Procedure for the Estimation of the Numerical Uncertainty of CFD Calculations Based on Grid Refinement Studies,” J. Comput. Phys., 262, pp. 104–130. [CrossRef]
Xing, T. , and Stern, F. , 2015, “ Comment on: ‘A Procedure for the Estimation of the Numerical Uncertainty of CFD Calculations Based on Grid Refinement Studies’ (Eça, L., and Hoekstra, M., 2014, Journal of Computational Physics, 262, pp. 104–130),” J. Comput. Phys., 301, pp. 484–486. [CrossRef]
Eça, L. , and Hoekstra, M. , 2015, “ Reply to Comment on ‘A Procedure for the Estimation of the Numerical Uncertainty of CFD Calculations Based on Grid Refinement Studies’ (L. Eça and M. Hoekstra, Journal of Computational Physics 262 (2014) 104–130),” J. Comput. Phys., 301, pp. 487–488. [CrossRef]
Phillips, T. , and Roy, C. , 2017, “ A New Extrapolation-Based Uncertainty Estimator for Computational Fluid Dynamics,” J. Verif., Validation Uncertainty Quantif., 1(4), p. 041006. [CrossRef]
Rider, W. , Witkowsk, W. , Kammc, J. R. , and Wildey, T. , 2016, “ Robust Verification Analysis,” J. Comput. Phys., 307, pp. 146–163. [CrossRef]
Abanto, J. , Pelletier, D. , Garon, A. , Trépanier, J.-Y. , and Reggio, M. , 2005, “Verification of Some Commercial CFD Codes on Atypical CFD Problems,” AIAA Paper No. 2005-682.
Hemsch, M. , 2004, “ Statistical Analysis of Computational Fluid Dynamics Solutions From the Drag Prediction Workshop,” J. Aircr., 41(1), pp. 95–103.
Vassberg, J. , 2016, “Challengers and Accomplishments of the AIAA CFD Drag Prediction Workshop Series,” AVT-246 Specialists' Meeting on Progress and Challenges in Validation Testing for Computational Fluid Dynamics, Avila, Spain, Sept. 26–28, Paper No. STO-MP-AVT-246 https://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwiX0JeKqLHXAhUD5CYKHd73D4YQFggsMAA&url=https%3A%2F%2Fwww.sto.nato.int%2Fpublications%2FSTO%2520Meeting%2520Proceedings%2FSTO-MP-AVT-246%2FMP-AVT-246-01.pdf&usg=AOvVaw2YpUxVa2CCr4xg3gsLvHZr.
Salas, M. , 2006, “ Digital Flight: The Last CFD Aeronautical Grand Challenge,” J. Sci. Comput., 28(2–3), p. 479.
Salas, M. , 2006, “ Some Observations on Grid Convergence,” Comput. Fluids, 35(7), pp. 688–692. [CrossRef]
Stern, F. , Wilson, R. , and Shao, J. , 2006, “ Quantitative Approach to V&V of CFD Simulations and Certification of CFD Codes With Examples,” Int. J. Numer. Methods Fluids, 50(11), pp. 1335–1355. [CrossRef]
Stern, F. , 2007, “ Quantitative V&V of CFD Solutions and Certification of CFD Codes,” AVT-147 Symposium on Computational Uncertainty in Military Vehicle Design, Athens, Greece, Paper No. RTO-MP-AVT-147 https://www.sto.nato.int/publications/STO%20Meeting%20Proceedings/RTO-MP-AVT-147/MP-AVT-147-21.pdf.
Larsson, L. , Stern, F. , and Bertram, V. , 2003, “ Benchmarking of Computational Fluid Dynamics for Ship Flows: The Gothenburg 2000 Workshop,” J. Ship Res., 47(1), pp. 63–81. http://www.sname.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=19df353f-82c9-44f5-9176-c601feff540b
Hino, T. , ed., 2005, CFD Workshop Tokyo 2005, National Maritime Research Institute, Tokyo, Japan.
Larsson, L. , Stern, F. , and Visonneau, M. , eds., 2014, Numerical Ship Hydrodynamics: An Assessment of the Gothenburg 2010 Workshop, Springer, Dordrecht, The Netherlands.
Stern, F. , Diez, M. , and Sadat-Hosseini, H. , 2016, “ Improved Statistical Approach for Certification of CFD Codes With Examples,” Verification and Validation Symposium, Las Vegas, NV, May 16–20, Paper No. VVS2016-8832.
Quadvlieg, F. , Simonsen, C. , Otzen, J. , and Stern, F. , 2015, “Review of the SIMMAN 2014 Workshop on the State of the Art of Prediction Techniques for Ship Maneuverability,” International Marine Simulator Forum (MARSIM Conference), Newcastle, UK, Sept. 8–11, p. 4.2.3.
Stern, F. , Agdrup, K. , Kim, S. Y. , Hochbaum, A. C. , Rhee, K. P. , Quadvlieg, F. , Perdon, P. , Hino, T. , Broglia, R. , and Gorski, J. , 2011, “ Experience From SIMMAN 2008: The First Workshop on Verification and Validation of Ship Maneuvering Simulation Methods,” J. Ship Res., 55(2), pp. 135–147. http://www.marin.nl/upload/25460dec-3220-44ec-a4b4-12b05502016d_2011-JSR-55-2-Stern_etal_Experience%20from%20SIMMAN%202008.pdf
Diez, M. , Broglia, R. , Durante, D. , Olivieri, A. , Campana, E. F. , and Stern, F. , 2017, “Validation of Uncertainty Quantification Methods for High-Fidelity CFD of Ship Response in Irregular Waves,” AIAA Paper No. 2017-1655.
Simonsen, C. D. , Otzen, J. F. , Nielson, C. , and Stern, F. , 2014, “ CFD Prediction of Added Resistance of the KCS in Regular Head and Oblique Waves,” 30th Symposium on Naval Hydrodynamics, Hobart, Australia, Nov. 2–7.
Quadvlieg, F. H. H. A. , and Brouwer, J. , 2011, “ KVLCC2 Benchmark Data Including Uncertainty Analysis to Support Manoeuvring Predictions,” Fourth International Conference on Computational Methods in Marine Engineering (ECCOMAS MARINE), Lisbon, Portugal, Sept. 28–30. http://www.marin.nl/web/Publications/Publication-items/KVLCC2-Benchmark-Data-Including-Uncertainty-Analysis-To-Support-Manoeuvring-Predictions.htm
ABS, 2006, “Guide for Vessel Maneuverability,” American Bureau of Shipping, Houston, TX.
IMO Resolution, 2002, “Standards for Ship Maneuverability,” International Maritime Organization, London, Standard No. MSC.137(76). http://www.imo.org/en/KnowledgeCentre/IndexofIMOResolutions/Maritime-Safety-Committee-(MSC)/Documents/MSC.137(76).pdf

Figures

Grahic Jump Location
Fig. 1

Benchmark hull forms and coordinate system: (a) KCS and (b) KVLCC2

Grahic Jump Location
Fig. 2

CT submissions: distribution of signed (a) and absolute (b) error; precision uncertainty of signed (c) and absolute (d) error; certification using signed error (e) and state-of-the-art assessment using absolute error

Grahic Jump Location
Fig. 3

Added resistance and motions: submission scatter, average submission, and experimental data (a), (d), (g), (j), and (m); absolute error scatter and state of the art assessment with (b), (e), (h), (k), and (n) and without (c), (f), (i), (l), and (o) outliers as per Chauvenet's criterion versus wavelength

Grahic Jump Location
Fig. 4

Definitions of the validation variables used in Table 6 for 20/20 zigzag maneuver. The first and second overshoot angles, α201 and α202 are the difference between the specified value 20 deg and the maximal heading angle reached before the course is reversed, the initial turning ability, ℓ20, is the distance that the vessel travels from the moment of the first execute (EX) of the rudder until the heading reaches 20 deg, the overshoot time, Tα201, is the time elapsed from the moment of the first EX to when the maximum change of heading is reached, and the period, P, is the time elapse between the second EX and the fourth EX. Although not shown in this figure, the variable rα201 is the maximum rate of the heading change before the first overshoot. The illustration is a modification of the American Bureau of Shipping guide for vessel maneuverability [41], Fig. 1 at page 27.

Grahic Jump Location
Fig. 5

KVLCC2 zigzag 20/20: (a) rudder angle, (b) heading, (c) yaw rate r, (d) drift angle, and (e) roll. The lines with the number/character symbols are the submissions as list in Table 5, where the number symbols “1” through “9” correspond to the entity numbers 1–9 and the character symbols “A,” “B,” and “C” to the entity numbers 10, 11, and 12, respectively. The bold black line is the experimental benchmark data, D. The bold line with error bars represents the average submission, S¯, where the error bars indicate the uncertainty limits of the average submission, US¯, at a set of selected time points. It is noted that the submission “7” was identified as an outlier and not included in the S¯ and US¯ calculations.

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In